From 8b40b2bb64cfed178e62c14093de6d6115a6206e Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 27 Aug 2018 09:51:26 -0700 Subject: [PATCH 1/7] A17: Client-Side Health Checking --- A17-client-side-health-checking.md | 315 +++++++++++++++++++++++++++++ 1 file changed, 315 insertions(+) create mode 100644 A17-client-side-health-checking.md diff --git a/A17-client-side-health-checking.md b/A17-client-side-health-checking.md new file mode 100644 index 000000000..390a583ef --- /dev/null +++ b/A17-client-side-health-checking.md @@ -0,0 +1,315 @@ +Client-Side Health Checking +---- +* Author(s): Mark D. Roth (roth@google.com) +* Approver: a11r +* Status: Draft +* Implemented in: +* Last updated: 2018-08-27 +* Discussion at: (filled after thread exists) + +## Abstract + +This document proposes a design for supporting application-level +health-checking on the client side. + +## Background + +gRPC has an existing [health-checking +mechanism](https://github.com/grpc/grpc/blob/master/doc/health-checking.md), +which allows server applications to signal that they are not healthy +without actually tearing down connections to clients. This is +used when (e.g.) a server is itself up but another service that it depends +on is not available. + +Currently, this mechanism is used in some [look-aside +load-balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md) +implementations to provide centralized health-checking of backend +servers from the balancer infrastructure. However, there is no +support for using this health-checking mechanism from the client side, +which is needed when not using look-aside load balancing (or when +falling back from look-aside load balancing to directly contacting +backends when the balancers are unreachable). + +### Related Proposals + +N/A + +## Proposal + +The gRPC client will be configured to send health-checking RPCs to +each backend that it is connected to. Whenever a backend responds as +unhealthy, the client's LB policy will stop sending requests to that +backend until it reports healthy again. + +Note that because the health-checking service requires a service name, the +client will need to be configured with a service name to use. However, +by default, it can use the empty string, which would mean that the +health of all services on a given host/port would be controlled with a +single switch. + +### Watch-Based Health Checking Protocol + +The current health-checking protocol is a unary API, where the client is +expected to periodically poll the server. That was sufficient for use +via UHC, where the health checks come from a small number of centralized +clients and where there is existing infrastructure to periodically +poll each client. However, if we are going to have large numbers of +clients issuing health-check requests, then we will need to convert the +health-checking protocol to a streaming Watch-based API for scalability +and bandwidth-usage reasons. + +Note that one down-side of this approach is that it could conceivably +be possible that the server-side health-checking code somehow fails +to send an update when becoming unhealthy. If the problem is due to +a server that stops polling for I/O, then the problem would be caught +by keepalives, at which point the client would disconnect. But if the +problem is caused by a bug in the health-checking service, then it's +possible that a server could still be responding but has failed to notify +the client that it is unhealthy. + +We can consider changing UHC to use this new streaming API, but that's +out of scope of this document. + +#### API Changes + +For the proposed API, see https://github.com/grpc/grpc-proto/pull/33. + +We propose to add the following RPC method to the health checking service: + +``` +rpc Watch(HealthCheckRequest) returns (stream HealthCheckResponse); +``` + +This is exactly the same as the existing `Check()` method, except that it +is server-side streaming instead of unary. The server will be expected +to send a message immediately upon receiving the message from the client +and will then be expected to send a new message whenever the requested +service's health changes. + +We will also add a new enum value to the `ServingStatus` enum: + +``` +SERVICE_UNKNOWN = 3; +``` + +This serving status will be returned by the `Watch()` method when the +requested service name is initially unknown by the server. Note that the +existing `Check()` unary method fails with status `NOT_FOUND` in this case, +and we are not proposing to change that behavior. + +### Client Behavior + +The health checking client code will be built into the subchannel, +so that individual LB policies do not need to explicitly add support +for health checking. (However, see also "LB Policies Can Disable Health +Checking When Needed" below.) + +Note that when an LB policy sees a subchannel go into state +`TRANSIENT_FAILURE`, it will not be able to tell whether the subchannel +has become disconnected or whether the backend has reported that it +is unhealthy. The LB policy may therefore take some unnecessary but +unharmful steps in response, such as asking the subchannel to connect +(which will be a no-op if it's already connected) or asking the resolver +to re-resolve (which will probably be throttled by the cooldown timer). + +#### Client Workflow + +When a subchannel first establishes a connection, if health checking +is enabled, the client will immediately start the `Watch()` call to the +backend. The subchannel will stay in state CONNECTING until it receives +the first health-check response from the backend (see "Race Conditions +and Subchannel Startup" below). + +When a client receives a health-checking response from the backend, +if the health check response indicates that the backend is healthy, the +subchannel will transition to state READY; otherwise, it will transition +to state `TRANSIENT_FAILURE`. Note that this means that when a backend +transitions from healthy to unhealthy, the subchannel's connectivity state +will transition from `TRANSIENT_FAILURE` directly to `READY`, with no stop at +CONNECTING in between. This is a new transition that was not [previously +supported](https://github.com/grpc/grpc/blob/master/doc/connectivity-semantics-and-api.md), although it is unlikely to be a problem for applications, +because (a) it won't affect the overall state of the parent channel +unless all subchannels are in the same state at the same time and (b) +because the connectivity state API can drop states anyway, the application +won't be able to tell that we didn't stop in `CONNECTING` in between anyway. + +If the `Watch()` call fails with status `UNIMPLEMENTED`, the client will +act as if health checking is disabled. That is, it will not retry +the health-checking call, but it will treat the channel as healthy +(connectivity state `READY`). However, the client will record a [channel +trace](https://github.com/grpc/proposal/blob/master/A3-channel-tracing.md) +event indicating that this has happened. + +If the `Watch()` call returns any other status, the subchannel will +transition to connectivity state `TRANSIENT_FAILURE` and will retry the +call. To avoid hammering a server that may be experiencing problems, +the client will use exponential backoff between attempts. When the client +receives a message from the server on a given call, the backoff state is +reset, so the next attempt will occur immediately, but any subsequent +attempt will be subject to exponential backoff. When the next attempt +starts, the subchannel will transition to state `CONNECTING`. + +When the client channel is shutting down or when the backend sends a +GOAWAY, the client will cancel the `Watch()` call. There is no need to +wait for the final status from the server in this case. + +#### Race Conditions and Subchannel Startup + +Note that because of the inherently asynchronous nature of the network, +whenever a backend transitions from healthy to unhealthy, it may still +receive a small number of RPCs that were already in flight from the +client before the client received the notification that the backend is +unhealthy. This race condition lasts approximately the one-way network +trip time (i.e., the time between when the backend sends the unhealthy +notification and when the client receives it). + +When the connection is first established, however, the problem is more +severe, because the client has not yet started the `Watch()` call, the race +condition actually lasts twice as long: it's not just the one-way network +trip time, but actually a full round trip (the client needs to start +the `Watch()` call, and the backend needs to send its initial response). + +To avoid this, the client will wait for the initial health-checking +response before the subchannel goes into state `READY`. However, this +does mean that when health checking is enabled, we require an additional +network RTT before the subchannel can be used. If this becomes a problem +for users that cannot be solved by simply disabling health-checking, +we can consider adding an option in the future to treat the subchannel +as healthy until the initial health-checking response is received. + +#### Call Credentials + +If there are any call credentials associated with the channel, the client +will send those credentials with the `Watch()` call. However, we will not +provide a way for the client to explicitly add call credentials for the +`Watch()` call. + +### Service Config Changes + +We will add the following new message to the service config: + +``` +message HealthCheckConfig { + // Service name to use in the health-checking request. + string service_name = 1; +} +``` + +We will then add the following top-level field to the ServiceConfig proto: + +``` +HealthCheckConfig health_check_config = 4; +``` + +Note that we currently need only this one parameter, but we are putting +it in its own message so that we have the option of adding more parameters +later if needed. + +### Defaults + +Client-side health checking will be disabled by default; users +will need to explicitly enable it via the service config when desired. + +### LB Policies Can Disable Health Checking When Needed + +There are some cases where an LB policy may want to disable client-side +health-checking. To allow this, we will provide a channel arg that +inhibits client-side health checks if it is enabled in the service config. + +This section details how each of our existing LB policies will interact +with health checking. + +#### `pick_first` + +We do not plan to support health checking with `pick_first`, for two +reasons. First, if you are only connecting to a single backend, you +probably don't want health-checking anyway, because if we don't send +RPCs to that backend, there's no where else for them to go, so they will +ultimately fail anyway. And second, it is not clear what the behavior +of `pick_first` would be for unhealthy channels; the naive approach of +treating an unhealthy channel as having failed and disconnecting would +both result in expensive reconnection attempts and might not actually +cause us to select a different backend after re-resolving. + +The `pick_first` LB policy will unconditionally set the channel arg to +inhibit health checking. + +#### `round_robin` + +Unhealthiness in `round_robin` will be handled in the obvious way: the +subchannel will be considered to not be in state `READY`, and picks will +not be assigned to it. + +#### `grpclb` + +The `grpclb` LB policy will set the channel arg to inhibit health checking +when we are using backend addresses obtained from the balancer, on the +assumption that the balancer is doing centralized health-checking. The arg +will not be set when using fallback addresses, since we want client-side +health checking for those. + +### Caveats + +Note that health checking will use a stream on every connection. +This stream counts toward the HTTP/2 `MAX_CONCURRENT_STREAMS` setting. +So, for example, any gRPC client or server that sets this to 1 will not +allow any other streams on the connection. + +Enabling health checks may affect the `MAX_CONNECTION_IDLE` setting. +We do not expect this to be a problem, since we are only implementing +client-side health checking in `round_robin`, which always attempts to +reconnect to every backend after an idle timeout anyway. + +## Rationale + +We discussed a large number of alternative approaches. The two most +appealling are listed here, along with the reasons why they were not +chosen. + +1. Use HTTP/2 SETTINGS type to indicate unhealthiness. Would avoid proto +dependency issues in core. However, would be transport-specific and would +not work for HTTP/2 proxies. Also would not allow signalling different +health status for multiple services on the same port. Would also +require server applications to use different APIs for the centralized +health checking case and the client-side health checking case. + +2. Have server stop listening when unhealthy. This would be challenging +to implement in Go due to the structure of the listener API. It would +also not allow signalling different health status for multiple services on +the same port. Would also require server applications to use different +APIs for the centralized health checking case and the client-side health +checking case. Would not allow individual clients to decide whether to +do health checking. In cases where the server's health was flapping up +and down, would cause a lot of overhead from connections being killed +and reestablished. + +## Implementation + +### C-core + +- Implement new `Watch()` method in health checking service + (https://github.com/grpc/grpc/pull/16351). +- Add support for health checking in subchannel code using nanopb + (in progress). + +### Java + +- Implement new `Watch()` method in health checking service. +- May need to redesign `IDLE_TIMEOUT`. Currently based on number of RPCs + on each transport (zero vs non-zero). Will need some way to exclude + the health-check RPC from this count. +- GRPC-LB needs to be aware of the setting to disable health checking on + its connections to backends. +- May need yet another runtime registry to find client health checking + implementation. Alternatively may hand-roll the protobuf + serialization. + +### Go + +- Implement new `Watch()` method in health checking service. +- Allow RPCs on a subchannel. +- Add mechanism in subconn for the health-check implementation, + optionally invoking it when subchannels are created, and affecting + connectivity state accordingly. +- Implement and register health checking in an optional package (to + enable/configure feature). From dba553be08b2987e5247591a85627da15da59f3c Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 27 Aug 2018 09:58:49 -0700 Subject: [PATCH 2/7] Add link to discussion thread. --- A17-client-side-health-checking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A17-client-side-health-checking.md b/A17-client-side-health-checking.md index 390a583ef..d062ffce7 100644 --- a/A17-client-side-health-checking.md +++ b/A17-client-side-health-checking.md @@ -5,7 +5,7 @@ Client-Side Health Checking * Status: Draft * Implemented in: * Last updated: 2018-08-27 -* Discussion at: (filled after thread exists) +* Discussion at: https://groups.google.com/d/topic/grpc-io/rwcNF4IQLlQ/discussion ## Abstract From b5e27279f43132f7481de9f73d04655b3b5baff2 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 27 Aug 2018 14:14:41 -0700 Subject: [PATCH 3/7] review changes --- A17-client-side-health-checking.md | 66 ++++++++++++++++-------------- 1 file changed, 35 insertions(+), 31 deletions(-) diff --git a/A17-client-side-health-checking.md b/A17-client-side-health-checking.md index d062ffce7..eb480a00a 100644 --- a/A17-client-side-health-checking.md +++ b/A17-client-side-health-checking.md @@ -51,12 +51,12 @@ single switch. The current health-checking protocol is a unary API, where the client is expected to periodically poll the server. That was sufficient for use -via UHC, where the health checks come from a small number of centralized -clients and where there is existing infrastructure to periodically -poll each client. However, if we are going to have large numbers of -clients issuing health-check requests, then we will need to convert the -health-checking protocol to a streaming Watch-based API for scalability -and bandwidth-usage reasons. +via centralized health checking via a look-aside load balancer, where +the health checks come from a small number of clients and where there +is existing infrastructure to periodically poll each client. However, +if we are going to have large numbers of clients issuing health-check +requests, then we will need to convert the health-checking protocol to +a streaming Watch-based API for scalability and bandwidth-usage reasons. Note that one down-side of this approach is that it could conceivably be possible that the server-side health-checking code somehow fails @@ -130,8 +130,8 @@ CONNECTING in between. This is a new transition that was not [previously supported](https://github.com/grpc/grpc/blob/master/doc/connectivity-semantics-and-api.md), although it is unlikely to be a problem for applications, because (a) it won't affect the overall state of the parent channel unless all subchannels are in the same state at the same time and (b) -because the connectivity state API can drop states anyway, the application -won't be able to tell that we didn't stop in `CONNECTING` in between anyway. +because the connectivity state API can drop states, the application won't +be able to tell that we didn't stop in `CONNECTING` in between anyway. If the `Watch()` call fails with status `UNIMPLEMENTED`, the client will act as if health checking is disabled. That is, it will not retry @@ -149,7 +149,7 @@ reset, so the next attempt will occur immediately, but any subsequent attempt will be subject to exponential backoff. When the next attempt starts, the subchannel will transition to state `CONNECTING`. -When the client channel is shutting down or when the backend sends a +When the client subchannel is shutting down or when the backend sends a GOAWAY, the client will cancel the `Watch()` call. There is no need to wait for the final status from the server in this case. @@ -157,17 +157,20 @@ wait for the final status from the server in this case. Note that because of the inherently asynchronous nature of the network, whenever a backend transitions from healthy to unhealthy, it may still -receive a small number of RPCs that were already in flight from the +receive some number of RPCs that were already in flight from the client before the client received the notification that the backend is -unhealthy. This race condition lasts approximately the one-way network -trip time (i.e., the time between when the backend sends the unhealthy -notification and when the client receives it). - -When the connection is first established, however, the problem is more -severe, because the client has not yet started the `Watch()` call, the race -condition actually lasts twice as long: it's not just the one-way network -trip time, but actually a full round trip (the client needs to start -the `Watch()` call, and the backend needs to send its initial response). +unhealthy. This race condition lasts approximately the network round +trip time (it includes the one-way trip time for RPCs that were already +sent by the client but not yet received by the server when the server +goes unhealthy, plus the one-way trip time for RPCs sent by the client +between when the server sends the unhealthy notification and when the +client receives it). + +When the connection is first established, however, the problem could +affect a larger number of RPCs, because there could be a number of RPCs +queued up waiting for the channel to become connected, which would all +be sent at once. And unlike an already-established connection, the race +condition is avoidable for newly established connections. To avoid this, the client will wait for the initial health-checking response before the subchannel goes into state `READY`. However, this @@ -207,29 +210,29 @@ later if needed. ### Defaults -Client-side health checking will be disabled by default; users +Client-side health checking will be disabled by default; service owners will need to explicitly enable it via the service config when desired. +There will be a channel argument that can be used on the client to +disable health checking even if it is enabled in the service config. + ### LB Policies Can Disable Health Checking When Needed There are some cases where an LB policy may want to disable client-side -health-checking. To allow this, we will provide a channel arg that -inhibits client-side health checks if it is enabled in the service config. +health-checking. To allow this, LB policies will be able to set the +channel argument mentioned above to inhibit health checking. This section details how each of our existing LB policies will interact with health checking. #### `pick_first` -We do not plan to support health checking with `pick_first`, for two -reasons. First, if you are only connecting to a single backend, you -probably don't want health-checking anyway, because if we don't send -RPCs to that backend, there's no where else for them to go, so they will -ultimately fail anyway. And second, it is not clear what the behavior -of `pick_first` would be for unhealthy channels; the naive approach of -treating an unhealthy channel as having failed and disconnecting would -both result in expensive reconnection attempts and might not actually -cause us to select a different backend after re-resolving. +We do not plan to support health checking with `pick_first`, because it +is not clear what the behavior of `pick_first` would be for unhealthy +channels. The naive approach of treating an unhealthy channel as having +failed and disconnecting would both result in expensive reconnection +attempts and might not actually cause us to select a different backend +after re-resolving. The `pick_first` LB policy will unconditionally set the channel arg to inhibit health checking. @@ -303,6 +306,7 @@ and reestablished. - May need yet another runtime registry to find client health checking implementation. Alternatively may hand-roll the protobuf serialization. +- Will need plumbing to allow creating a trace event. ### Go From 0c640848e3f1227c3eacd78c1bfc27a6841732ea Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 27 Aug 2018 14:23:19 -0700 Subject: [PATCH 4/7] Fix backwardsness. --- A17-client-side-health-checking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A17-client-side-health-checking.md b/A17-client-side-health-checking.md index eb480a00a..0cf5a0a03 100644 --- a/A17-client-side-health-checking.md +++ b/A17-client-side-health-checking.md @@ -124,7 +124,7 @@ When a client receives a health-checking response from the backend, if the health check response indicates that the backend is healthy, the subchannel will transition to state READY; otherwise, it will transition to state `TRANSIENT_FAILURE`. Note that this means that when a backend -transitions from healthy to unhealthy, the subchannel's connectivity state +transitions from unhealthy to healthy, the subchannel's connectivity state will transition from `TRANSIENT_FAILURE` directly to `READY`, with no stop at CONNECTING in between. This is a new transition that was not [previously supported](https://github.com/grpc/grpc/blob/master/doc/connectivity-semantics-and-api.md), although it is unlikely to be a problem for applications, From d459791d0b071ea507144ed022f4eb8072614d7d Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 27 Aug 2018 14:41:34 -0700 Subject: [PATCH 5/7] Change service config to use google.protobuf.StringValue instead of string. --- A17-client-side-health-checking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A17-client-side-health-checking.md b/A17-client-side-health-checking.md index 0cf5a0a03..ef6a813dc 100644 --- a/A17-client-side-health-checking.md +++ b/A17-client-side-health-checking.md @@ -194,7 +194,7 @@ We will add the following new message to the service config: ``` message HealthCheckConfig { // Service name to use in the health-checking request. - string service_name = 1; + google.protobuf.StringValue service_name = 1; } ``` From dbd6520a4d5c8a43ccbd439ce8b0c2c6e836a542 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 27 Sep 2018 08:31:24 -0700 Subject: [PATCH 6/7] Addressed review comments from @a11r. --- A17-client-side-health-checking.md | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/A17-client-side-health-checking.md b/A17-client-side-health-checking.md index ef6a813dc..7df27ef55 100644 --- a/A17-client-side-health-checking.md +++ b/A17-client-side-health-checking.md @@ -36,16 +36,18 @@ N/A ## Proposal -The gRPC client will be configured to send health-checking RPCs to -each backend that it is connected to. Whenever a backend responds as -unhealthy, the client's LB policy will stop sending requests to that +The gRPC client will be able to be configured to send health-checking RPCs +to each backend that it is connected to. Whenever a backend responds +as unhealthy, the client's LB policy will stop sending requests to that backend until it reports healthy again. Note that because the health-checking service requires a service name, the client will need to be configured with a service name to use. However, by default, it can use the empty string, which would mean that the health of all services on a given host/port would be controlled with a -single switch. +single switch. Semantically, the empty string is used to represent the +overall health of the server, as opposed to the health of any individual +services running on the server. ### Watch-Based Health Checking Protocol @@ -67,9 +69,6 @@ problem is caused by a bug in the health-checking service, then it's possible that a server could still be responding but has failed to notify the client that it is unhealthy. -We can consider changing UHC to use this new streaming API, but that's -out of scope of this document. - #### API Changes For the proposed API, see https://github.com/grpc/grpc-proto/pull/33. @@ -99,6 +98,12 @@ and we are not proposing to change that behavior. ### Client Behavior +Client-side health checking will be disabled by default; +service owners will need to explicitly enable it via the [service +config](#service-config-changes) when desired. There will be a channel +argument that can be used on the client to disable health checking even +if it is enabled in the service config. + The health checking client code will be built into the subchannel, so that individual LB policies do not need to explicitly add support for health checking. (However, see also "LB Policies Can Disable Health @@ -138,7 +143,8 @@ act as if health checking is disabled. That is, it will not retry the health-checking call, but it will treat the channel as healthy (connectivity state `READY`). However, the client will record a [channel trace](https://github.com/grpc/proposal/blob/master/A3-channel-tracing.md) -event indicating that this has happened. +event indicating that this has happened. It will also log a message at +priority ERROR. If the `Watch()` call returns any other status, the subchannel will transition to connectivity state `TRANSIENT_FAILURE` and will retry the @@ -208,14 +214,6 @@ Note that we currently need only this one parameter, but we are putting it in its own message so that we have the option of adding more parameters later if needed. -### Defaults - -Client-side health checking will be disabled by default; service owners -will need to explicitly enable it via the service config when desired. - -There will be a channel argument that can be used on the client to -disable health checking even if it is enabled in the service config. - ### LB Policies Can Disable Health Checking When Needed There are some cases where an LB policy may want to disable client-side From f435179abbd4d96ad80d88c01a8ed073a3fdba67 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 1 Oct 2018 08:30:52 -0700 Subject: [PATCH 7/7] Added link to service config doc and example in JSON form. --- A17-client-side-health-checking.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/A17-client-side-health-checking.md b/A17-client-side-health-checking.md index 7df27ef55..40394daea 100644 --- a/A17-client-side-health-checking.md +++ b/A17-client-side-health-checking.md @@ -195,7 +195,8 @@ provide a way for the client to explicitly add call credentials for the ### Service Config Changes -We will add the following new message to the service config: +We will add the following new message to the [service +config](https://github.com/grpc/grpc/blob/master/doc/service_config.md): ``` message HealthCheckConfig { @@ -214,6 +215,13 @@ Note that we currently need only this one parameter, but we are putting it in its own message so that we have the option of adding more parameters later if needed. +Here is an example of how one would set the health checking service name +to "foo" in the service config in JSON form: + +``` +"healthCheckConfig": {"serviceName": "foo"} +``` + ### LB Policies Can Disable Health Checking When Needed There are some cases where an LB policy may want to disable client-side