From 4a25ec60898d9be8a3bce1c71423c1dba1564510 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 31 Jan 2017 10:19:09 -0800 Subject: [PATCH 1/7] Proposal for encoding grpclb data in DNS. --- A5.md | 157 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100644 A5.md diff --git a/A5.md b/A5.md new file mode 100644 index 000000000..3b2eda6c8 --- /dev/null +++ b/A5.md @@ -0,0 +1,157 @@ +Load Balancing and DNS +---- +* Author(s): Mark D. Roth (roth@google.com) +* Approver: a11r +* Status: Draft +* Implemented in: +* Last updated: 2017-01-31 +* Discussion at: (filled after thread exists) + +## Abstract + +This document describes how additional name service data for load +balancing should be encoded in DNS. + +## Background + +As described in the [name +resolution](https://github.com/grpc/grpc/blob/master/doc/naming.md) doc, +a resolver implementation is expected to return two additional pieces of +data with each address, both related to [load +balancing](https://github.com/grpc/grpc/blob/master/doc/load-balancing.md). +Those two pieces of data are: + +- A boolean indicating whether the address is a backend address (i.e., + the address to use to contact the server directly) or a balancer + address. + +- The name of the balancer, if the address is a balancer address. + This will be used to perform peer authorization. + +This document describes how those two pieces of data should be encoded +when the name service is DNS. + +### Related Proposals: + +N/A + +## Proposal + +This proposal makes use of DNS SRV records, as described in +[RFC-2782](https://tools.ietf.org/html/rfc2782). The SRV record for +grpclb will use the following values: + +- service name: `grpclb` +- protocol: `tcp` +- priority: this will be `0` for all grpclb SRV records +- weight: this will be `0` for all grpclb SRV records + +When the gRPC client library opens a channel for a given server name, +it will ask DNS for these SRV records in addition to the normal address +records. For each SRV record that comes back in the response, the +client library will then do a DNS lookup for address records for the name +specified by the SRV record. It will then return each of the resulting +addresses with the extra fields indicating that the addresses are +balancer addresses and the name from the SRV record. + +### Example + +For example, let's say that we have the following DNS zone file: + +``` +$ORIGIN example.com. +; grpclb for server.example.com goes to lb.example.com:1234 +_grpclb._tcp.server IN SRV 0 0 1234 lb +; lb.example.com has 3 IP addresses +lb IN A 10.0.0.1 + IN A 10.0.0.2 + IN A 10.0.0.3 +``` + +With this data, a gRPC client will resolve the name `server.example.com` +to the following addresses: + +``` +address=10.0.0.1:1234, is_balancer=true, balancer_name=lb.example.com +address=10.0.0.2:1234, is_balancer=true, balancer_name=lb.example.com +address=10.0.0.3:1234, is_balancer=true, balancer_name=lb.example.com +``` + +### Future Work: Providing Both Server and Balancer Results + +For redundancy, it may be desirable to return both server and balancer +addresses for the same server name. In principle, this will allow the +client to fall back to directly contacting the servers if the load +balancers are unreachable. However, that fallback functionality is not +yet implemented, so it will be the subject of future work. + +However, it is still possible for service owners to set up their DNS +zone files to publish both types of addresses, so that they are prepared +for an eventual future when this fallback functionality has been +implemented. For example: + +``` +$ORIGIN example.com. +server IN A 10.0.0.11 + IN A 10.0.0.12 +; grpclb for server.example.com goes to lb.example.com:1234 +_grpclb._tcp.server IN SRV 0 0 1234 lb +; lb.example.com has 3 IP addresses +lb IN A 10.0.0.1 + IN A 10.0.0.2 + IN A 10.0.0.3 +``` + +For now, if both address records and SRV records are present for the +same server name, the gRPC client will ignore the address records and +return only the SRV records -- in other words, the result for a lookup +of `server.example.com` will be exactly the same in this case as it +would be in the previous example. + +In the future, when we do implement support for this kind of fallback, +this lookup will result in the following addresses: + +``` +address=10.0.0.11:443, is_balancer=false, balancer_name= +address=10.0.0.12:443, is_balancer=false, balancer_name= +address=10.0.0.1:1234, is_balancer=true, balancer_name=lb.example.com +address=10.0.0.2:1234, is_balancer=true, balancer_name=lb.example.com +address=10.0.0.3:1234, is_balancer=true, balancer_name=lb.example.com +``` + +Note that port 443 is the default if not specified by the server name +passed to the client library by the application. If the server name +passed in by the application does specify a port, that would be used for +the resulting server addresses, but it would not affect the port used +for the balancer addresses. + +## Rationale + +We considered one alternative, which was adding additional fields to a +TXT record instead of using an SRV record. However, the SRV record +seems to be a much better fit for this purpose. + +## Implementation + +### C-core + +In C-core, this implementation will rely on use of the c-ares DNS +library, which is being added in +[grpc/grpc#7771](https://github.com/grpc/grpc/pull/7771). + +Specifically, we will change the c-ares resolver implementation to +automatically look up the grpclb SRV records for the specified name. If +found, it will then look up the name(s) that the SRV records point to +and return those as balancer addresses. + +### Java + +TODO(notcarl): Provide content for this section. + +### Go + +TODO(roth): Talk to Go folks about content for this section. + +## Open issues (if applicable) + +N/A From 51c60479da83857643cc01e8a0029456f7cd5497 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 31 Jan 2017 12:25:40 -0800 Subject: [PATCH 2/7] Add link to discussion thread. --- A5.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A5.md b/A5.md index 3b2eda6c8..3cb6b9668 100644 --- a/A5.md +++ b/A5.md @@ -5,7 +5,7 @@ Load Balancing and DNS * Status: Draft * Implemented in: * Last updated: 2017-01-31 -* Discussion at: (filled after thread exists) +* Discussion at: https://groups.google.com/d/topic/grpc-io/6be1QsHyZkk/discussion ## Abstract From 0c4cb1431911bca190a706629c6c5cc1e31ad707 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 2 Feb 2017 08:42:08 -0800 Subject: [PATCH 3/7] Add note about initially not supporting this for Windows or Node. --- A5.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/A5.md b/A5.md index 3cb6b9668..4035f069d 100644 --- a/A5.md +++ b/A5.md @@ -144,6 +144,10 @@ automatically look up the grpclb SRV records for the specified name. If found, it will then look up the name(s) that the SRV records point to and return those as balancer addresses. +Note that, due to platform support issues, we will initially *not* +support the c-ares resolver under Windows or for Node. Alternatives +will need to be found for these environments. + ### Java TODO(notcarl): Provide content for this section. From cde3d86a3432b678a714a8f47cac72da44632aca Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Thu, 23 Feb 2017 07:55:58 -0800 Subject: [PATCH 4/7] Add paragraph about priority and weight fields. --- A5.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/A5.md b/A5.md index 4035f069d..5aaef11b1 100644 --- a/A5.md +++ b/A5.md @@ -3,8 +3,8 @@ Load Balancing and DNS * Author(s): Mark D. Roth (roth@google.com) * Approver: a11r * Status: Draft -* Implemented in: -* Last updated: 2017-01-31 +* Implemented in: +* Last updated: 2017-02-23 * Discussion at: https://groups.google.com/d/topic/grpc-io/6be1QsHyZkk/discussion ## Abstract @@ -54,6 +54,15 @@ specified by the SRV record. It will then return each of the resulting addresses with the extra fields indicating that the addresses are balancer addresses and the name from the SRV record. +Note that the gRPC library will ignore the values of the `priority` and +`weight` fields. Instead, it will put all balancer addresses into a +single list (in whatever order the DNS server returns them) and use the +`pick_first` load-balancing policy to decide which one of them to use. +However, in order to leave open the possibility of adding support for +priority and weight in the future, we recommend that these fields be set +to 0. That way, if we do add support for these fields in the future, +existing records will not break. + ### Example For example, let's say that we have the following DNS zone file: From 0999e720b47ffe58a4a105ab47316054790bf741 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 14 Mar 2017 10:14:38 -0700 Subject: [PATCH 5/7] Renamed as per https://github.com/grpc/proposal/pull/17 --- A5.md => A5-grpclb-in-dns.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename A5.md => A5-grpclb-in-dns.md (100%) diff --git a/A5.md b/A5-grpclb-in-dns.md similarity index 100% rename from A5.md rename to A5-grpclb-in-dns.md From 7dcb6d00c0e7b319b3982bfd84366c69dbc6913e Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Mon, 1 May 2017 09:19:03 -0700 Subject: [PATCH 6/7] Added java implementation plan. --- A5-grpclb-in-dns.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/A5-grpclb-in-dns.md b/A5-grpclb-in-dns.md index 5aaef11b1..a56a86b9d 100644 --- a/A5-grpclb-in-dns.md +++ b/A5-grpclb-in-dns.md @@ -4,7 +4,7 @@ Load Balancing and DNS * Approver: a11r * Status: Draft * Implemented in: -* Last updated: 2017-02-23 +* Last updated: 2017-05-01 * Discussion at: https://groups.google.com/d/topic/grpc-io/6be1QsHyZkk/discussion ## Abstract @@ -159,7 +159,10 @@ will need to be found for these environments. ### Java -TODO(notcarl): Provide content for this section. +Java will depend on JNDI, Netty DNS, dnsjava, or a another DNS library +to do SRV record resolution. The existing name resolver, `DnsNameResolver`, +will be modified to resolve the additional records and include them in +the Attributes presented to the load balancer. ### Go From a06f9816bd2fd5e14f689186902fb93ce9581550 Mon Sep 17 00:00:00 2001 From: "Mark D. Roth" Date: Tue, 16 May 2017 09:21:30 -0700 Subject: [PATCH 7/7] Add go implementation information. --- A5-grpclb-in-dns.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/A5-grpclb-in-dns.md b/A5-grpclb-in-dns.md index a56a86b9d..decb28a0d 100644 --- a/A5-grpclb-in-dns.md +++ b/A5-grpclb-in-dns.md @@ -4,7 +4,7 @@ Load Balancing and DNS * Approver: a11r * Status: Draft * Implemented in: -* Last updated: 2017-05-01 +* Last updated: 2017-05-16 * Discussion at: https://groups.google.com/d/topic/grpc-io/6be1QsHyZkk/discussion ## Abstract @@ -145,7 +145,7 @@ seems to be a much better fit for this purpose. ### C-core In C-core, this implementation will rely on use of the c-ares DNS -library, which is being added in +library, which was added in [grpc/grpc#7771](https://github.com/grpc/grpc/pull/7771). Specifically, we will change the c-ares resolver implementation to @@ -166,7 +166,8 @@ the Attributes presented to the load balancer. ### Go -TODO(roth): Talk to Go folks about content for this section. +Go will use the `LookupSRV` function in package `net` to do SRV record +resolution. ## Open issues (if applicable)