Merge pull request kubernetes-sigs#20 from salrashid123/add-grpc-lb-e…

…xample Add e2e example for gRPC LB
jaison-tiu · Jun 7, 2021 · 965ce02 · 965ce02
2 parents 88ac7e6 + 1181d6d
commit 965ce02
Show file tree

Hide file tree

Showing 8 changed files with 441 additions and 16 deletions.
diff --git a/ingress/ingress-custom-grpc-health-check/README.md b/ingress/ingress-custom-grpc-health-check/README.md
@@ -38,27 +38,37 @@ We will provide example YAMLs for both cases, as well as describing nuances in t
 
 Below is an example Deployment specification when using a health check proxy. You may use the image we have referenced below or write your own. Full documentation on our provided image can be found [here](https://github.com/salrashid123/grpc_health_proxy).
 
-```
-spec:
- containers:
- - name: hc-proxy
-   image: docker.io/salrashid123/grpc_health_proxy
-   args: [
-     "--http-listen-addr=localhost:8081",
-     "--grpc-addr=localhost:8080",
-     "--service-name=echo.EchoServer",
-   ]
-   ports:
-   - containerPort: 8081
- - name: grpc-app
-   image: gcr.io/mygcr/myapp
-   ports:
-   - containerPort: 8080
+```yaml
+    spec:
+      containers:
+      - name: hc-proxy
+        image: gcr.io/cloud-solutions-images/grpc_health_proxy:1.0.0
+        args: [
+          "--http-listen-addr=0.0.0.0:8080",
+          "--grpcaddr=localhost:50051",
+          "--service-name=echo.EchoServer",
+          "--https-listen-ca=/config/CA_crt.pem",
+          "--https-listen-cert=/certs/http_server_crt.pem",
+          "--https-listen-key=/certs/http_server_key.pem",
+          "--grpctls",        
+          "--grpc-sni-server-name=grpc.domain.com",
+          "--grpc-ca-cert=/config/CA_crt.pem",
+          "--logtostderr=1",
+          "-v=1"
+        ]
+        ports:
+        - containerPort: 8080        
+      - name: grpc-app
+        image: gcr.io/mygcr/grpc_app
+        ports:
+        - containerPort: 50051        
 ```
 
 Remember that your gRPC application must implement the gRPC health protocol
 (grpc.health.v1.Health).
 
+NOTE: both the HTTP HealthCheck proxy and the gRPC Container should support TLS
+
 #### Service & BackendConfig
 
 For InstanceGroup backends, the port used for the health check must be exposed in the Service specification. This is needed so that a NodePort can be allocated. This NodePort is then applied to the BackendConfig resource (See below)
@@ -84,6 +94,8 @@ spec:
     requestPath: [PATH]
 ```
 
+For a full end-to-end example, see the `example/` folder.
+
 ### Non-proxied Health Check
 
 ```

diff --git a/ingress/ingress-custom-grpc-health-check/example/README.md b/ingress/ingress-custom-grpc-health-check/example/README.md
@@ -0,0 +1,129 @@
+### Example GKE Deployment
+
+For an end-to-end example of gRPC Load Balancing with gRPC:
+
+First deploy a GKE cluster with NEG enabled:
+
+```bash
+$ gcloud container  clusters create cluster-1 \
+  --machine-type "n1-standard-2" \
+  --zone us-central1-a  --num-nodes 2 --enable-ip-alias  -q
+```
+
+Deploy application
+
+```bash
+kubectl apply -f .
+```
+
+Wait ~8mins and note down the external and ILB addresses
+
+```bash
+$ kubectl get po,svc,ing
+NAME                                READY   STATUS    RESTARTS   AGE
+pod/fe-deployment-6c96c9648-sztpp   2/2     Running   0          112s
+pod/fe-deployment-6c96c9648-zj659   2/2     Running   0          119s
+
+NAME                     TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
+service/fe-srv-ingress   ClusterIP   10.10.44.25   <none>        50051/TCP   3m
+service/kubernetes       ClusterIP   10.10.32.1    <none>        443/TCP     4d9h
+
+NAME                                       CLASS    HOSTS   ADDRESS         PORTS     AGE
+ingress.networking.k8s.io/fe-ilb-ingress   <none>   *       10.128.0.77     80, 443   3m1s
+ingress.networking.k8s.io/fe-ingress       <none>   *       34.120.140.72   80, 443   3m1s
+
+export XLB_IP=`kubectl get ingress.extensions/fe-ingress -o jsonpath='{.status.loadBalancer.ingress[].ip}'`
+export ILB_IP=`kubectl get ingress.extensions/fe-ilb-ingress -o jsonpath='{.status.loadBalancer.ingress[].ip}'`
+```
+
+
+#### Test External
+
+Verify external loadbalancing by transmitting 10 RPCs over one channel.  The responses will show different pods that handled each request
+
+```log
+$ docker run --add-host grpc.domain.com:$XLB_IP  \
+  -t gcr.io/cloud-solutions-images/grpc_app /grpc_client \
+   --host=grpc.domain.com:443 --tlsCert /certs/CA_crt.pem \
+   --servername grpc.domain.com --repeat 10
+
+2021/01/27 12:53:08 RPC HealthChekStatus:SERVING
+2021/01/27 12:53:08 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:53:08 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+2021/01/27 12:53:08 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+2021/01/27 12:53:09 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:53:09 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+2021/01/27 12:53:09 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:53:09 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+2021/01/27 12:53:09 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+2021/01/27 12:53:09 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:53:09 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+```
+
+Now scale the number of pods
+
+```bash
+$ kubectl scale --replicas=10 deployment.apps/fe-deployment
+```
+
+```bash
+$ kubectl get po
+NAME                            READY   STATUS    RESTARTS   AGE
+fe-deployment-6c96c9648-2c9vv   2/2     Running   0          18s
+fe-deployment-6c96c9648-9p7q9   2/2     Running   0          18s
+fe-deployment-6c96c9648-d4rx2   2/2     Running   0          18s
+fe-deployment-6c96c9648-gpjvg   2/2     Running   0          18s
+fe-deployment-6c96c9648-h7p2c   2/2     Running   0          18s
+fe-deployment-6c96c9648-s9tl2   2/2     Running   0          18s
+fe-deployment-6c96c9648-sztpp   2/2     Running   0          4m32s
+fe-deployment-6c96c9648-tmmwd   2/2     Running   0          18s
+fe-deployment-6c96c9648-xc7d7   2/2     Running   0          18s
+fe-deployment-6c96c9648-zj659   2/2     Running   0          4m39s
+```
+
+Rerun the test.  Notice the new pods in the response 
+```log
+$ docker run --add-host grpc.domain.com:$XLB_IP \
+   -t gcr.io/cloud-solutions-images/grpc_app /grpc_client \
+   --host=grpc.domain.com:443 --tlsCert /certs/CA_crt.pem  \
+   --servername grpc.domain.com --repeat 10
+
+2021/01/27 12:54:18 RPC HealthChekStatus:SERVING
+2021/01/27 12:54:19 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:54:19 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-xc7d7"
+2021/01/27 12:54:19 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-d4rx2"
+2021/01/27 12:54:19 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-s9tl2"
+2021/01/27 12:54:19 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-2c9vv"
+2021/01/27 12:54:19 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+2021/01/27 12:54:19 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:54:19 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:54:19 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-h7p2c"
+2021/01/27 12:54:19 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+```
+
+#### Test Internal
+
+To test the internal loadbalancer, you must configure a VM from within an [allocated network](https://cloud.google.com/load-balancing/docs/l7-internal/setting-up-l7-internal#configuring_the_proxy-only_subnet) and export the environment variable `$XLB_IP` locally
+
+```log
+ $ docker run --add-host grpc.domain.com:$XLB_IP \
+     -t gcr.io/cloud-solutions-images/grpc_app /grpc_client \
+     --host=grpc.domain.com:443 --tlsCert /certs/CA_crt.pem  \
+     --servername grpc.domain.com --repeat 10
+
+2021/01/27 12:51:45 RPC HealthChekStatus:SERVING
+2021/01/27 12:51:45 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:51:45 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+2021/01/27 12:51:45 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:51:45 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+2021/01/27 12:51:45 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:51:45 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+2021/01/27 12:51:45 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:51:45 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+2021/01/27 12:51:45 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-sztpp"
+2021/01/27 12:51:45 RPC Response: message:"Hello unary RPC msg   from hostname fe-deployment-6c96c9648-zj659"
+```
+
+Source images used in this example can be found here:
+  - [gcr.io/cloud-solutions-images/grpc_health_proxy](https://github.com/salrashid123/grpc_health_proxy)
+  - [gcr.io/cloud-solutions-images/grpc_app](https://github.com/salrashid123/grpc_health_proxy/tree/master/example)
diff --git a/ingress/ingress-custom-grpc-health-check/example/fe-configmap.yaml b/ingress/ingress-custom-grpc-health-check/example/fe-configmap.yaml
@@ -0,0 +1,65 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: settings
+data:
+  CA_crt.pem:  |
+    -----BEGIN CERTIFICATE-----
+    MIIEDTCCAvWgAwIBAgIBAjANBgkqhkiG9w0BAQUFADBQMQswCQYDVQQGEwJVUzEP
+    MA0GA1UECgwGR29vZ2xlMRMwEQYDVQQLDApFbnRlcnByaXNlMRswGQYDVQQDDBJF
+    bnRlcnByaXNlIFJvb3QgQ0EwHhcNMjAwNDI1MjIwNzE1WhcNMzAwNDI1MjIwNzE1
+    WjBXMQswCQYDVQQGEwJVUzEPMA0GA1UECgwGR29vZ2xlMRMwEQYDVQQLDApFbnRl
+    cnByaXNlMSIwIAYDVQQDDBlFbnRlcnByaXNlIFN1Ym9yZGluYXRlIENBMIIBIjAN
+    BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2FXMxoPucnmPbeaYvtC7mwqMqS/D
+    RGuGmk78f8cLLmtf4qcfQwS5t5UkEAUiHaDGTdkk8AWRVGc4Sh/nwxlsEMw7thkl
+    W3zge8cU7V+pWjGokaYiLPqayV3CJ7VpYPIbXRGqeXQrSAh3h5vPmFn2IN4TZF2E
+    46Fob8xhqjYc9CAGqh8NevCyvyNvnb2ZTzQeC2jouRStltdHh97ynK/iatyzyot0
+    +9BrhI/9CBELS1MDGxcT35g48pEJzHr1/k3Wdz2VM0+pKSIBhiJM1t4Q1LALhP7L
+    KSr6Ex3H1OzaBEW8gpIoKlkH6I1D9lOs1rSSJU4ZEsVqft+3j32SByZNmQIDAQAB
+    o4HqMIHnMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1Ud
+    DgQWBBS/4RzwIkiP/DvPXdntrohwId/dhjAfBgNVHSMEGDAWgBSaVNYZIVg6RrbU
+    rgV+gHD0OM8oDDBFBggrBgEFBQcBAQQ5MDcwNQYIKwYBBQUHMAKGKWh0dHA6Ly9w
+    a2kuZXNvZGVtb2FwcDIuY29tL2NhL3Jvb3QtY2EuY2VyMDoGA1UdHwQzMDEwL6At
+    oCuGKWh0dHA6Ly9wa2kuZXNvZGVtb2FwcDIuY29tL2NhL3Jvb3QtY2EuY3JsMA0G
+    CSqGSIb3DQEBBQUAA4IBAQCIK7eSGxf7Vy/5pKVdDGgD/sG/pjBgqP6IVocvLUN0
+    nUzT8NMkOYraJz+LySUy55WGPeFeARcRCqeFwqz+o1BetdqOUZgouf1DuXvfyX0w
+    cp1Y/IEPcFnzl+U5b1F10iOLGt5llUxMd6eo1MS53SrbF3NIvQOu6aoHBlh614qk
+    zcx2hAhLIw5C9MvK7r5UP179ilv4x7ZYfWsM7DLvhgG8dEgQTDCPB5h6cSKIM1qI
+    hHzpPkrGcGtKK/Fbidjw4OP25HADV6i/GXKkjirOhVQkoj2S3RjkiNy8JvJobCFM
+    GJSi6m9GO8OrUA+o9r9JKYkLmLEWD5PeF7hCnbGQY+Cw
+    -----END CERTIFICATE-----
+    -----BEGIN CERTIFICATE-----
+    MIIDfjCCAmagAwIBAgIBATANBgkqhkiG9w0BAQUFADBQMQswCQYDVQQGEwJVUzEP
+    MA0GA1UECgwGR29vZ2xlMRMwEQYDVQQLDApFbnRlcnByaXNlMRswGQYDVQQDDBJF
+    bnRlcnByaXNlIFJvb3QgQ0EwHhcNMjAwNDI1MjIwNDQxWhcNMzAwNDI1MjIwNDQx
+    WjBQMQswCQYDVQQGEwJVUzEPMA0GA1UECgwGR29vZ2xlMRMwEQYDVQQLDApFbnRl
+    cnByaXNlMRswGQYDVQQDDBJFbnRlcnByaXNlIFJvb3QgQ0EwggEiMA0GCSqGSIb3
+    DQEBAQUAA4IBDwAwggEKAoIBAQDxuasMnn/6NdpXZk9FnWE109YIUDXdPyN7Qiq1
+    mX54dBzoC/vZ1mlLyATGe9+DF9cgRa1apQAlid4cUsazo1Tropivf+y/gt9/6mXO
+    X0zEAZgwnw4VU6NXAVpyzMFiV8kFO18lzUlXdt92SGRiS6Bpk0oxuNZBYKx8SUDY
+    U7veKF2Gz9GcF5MXUQ5VxbUPqvFcuKGR4DRFAYUQWhB+c2XWr3Z7ibap5HxSRFJl
+    TZnE7GdeowwzV8/k2bmRWTznhJrGbnXRUykBF+Jhs29BRfqlKroJqL8MZ4VS9j+J
+    fQRIxYkG4JxfSOuL3mELJQUvZfZADMb60dPrMqJQeItBDndJAgMBAAGjYzBhMA4G
+    A1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSaVNYZIVg6
+    RrbUrgV+gHD0OM8oDDAfBgNVHSMEGDAWgBSaVNYZIVg6RrbUrgV+gHD0OM8oDDAN
+    BgkqhkiG9w0BAQUFAAOCAQEAJHWSDi4Ku1L2h1L3Xz3EvDfARTBkBpMkTylQQ/yW
+    Y14CNso7cpZMnEmHhpscZHXzfFIKQgQXhSV4UeF1qSsVau5n37Kpf93XV/i8zL33
+    EAm/1HwrMCfVdBLx8pAj6GmuigPAnO5wH9B+KMolVyp91fptilmVzznp/KdMzurl
+    BqDjlLCt8eWzZ0WTmJhc0nWfgdwhlKD5DeUe6tmpahZK3ls3j1/F62FuccwJioem
+    krx0EqCm02vc8QU/r7OAEEgenmmak+yrKz4MV3LtdzO+R0F7/OMObepAE1ppCaP4
+    3Nh6qNxO8UxfBjkfxnGEkp9VicG96HSplK/L3CHYHLvKzQ==
+    -----END CERTIFICATE-----
diff --git a/ingress/ingress-custom-grpc-health-check/example/fe-deployment.yaml b/ingress/ingress-custom-grpc-health-check/example/fe-deployment.yaml
@@ -0,0 +1,82 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: fe-deployment
+  labels:
+    app: fe
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: fe
+  template:
+    metadata:
+      labels:
+        app: fe
+    spec:
+      containers:
+      - name: hc-proxy
+        image: gcr.io/cloud-solutions-images/grpc_health_proxy:1.0.0
+        args: [
+          "--http-listen-addr=0.0.0.0:8443",
+          "--grpcaddr=localhost:50051",
+          "--service-name=echo.EchoServer",
+          "--https-listen-ca=/config/CA_crt.pem",
+          "--https-listen-cert=/certs/http_server_crt.pem",
+          "--https-listen-key=/certs/http_server_key.pem",
+          "--grpctls",        
+          "--grpc-sni-server-name=grpc.domain.com",
+          "--grpc-ca-cert=/config/CA_crt.pem",
+          "--logtostderr=1",
+          "-v=1"
+        ]
+        ports:
+        - containerPort: 8443
+        volumeMounts:
+        - name: config-vol
+          mountPath: /config
+          readOnly: true
+        - name: certs-vol
+          mountPath: /certs
+          readOnly: true          
+      - name: grpc-app
+        image: gcr.io/cloud-solutions-images/grpc_app
+        args: [
+          "/grpc_server",
+          "--grpcport=0.0.0.0:50051",
+          "--tlsCert=/certs/tls.crt",
+          "--tlsKey=/certs/tls.key"        
+        ]
+        ports:
+        - containerPort: 50051
+        volumeMounts:
+          - name: fe-certs-vol
+            mountPath: /certs
+            readOnly: true          
+      volumes:
+      - name: config-vol
+        configMap:
+          name: settings
+          items:
+          - key: CA_crt.pem
+            path: CA_crt.pem
+      - name: certs-vol
+        secret:
+          secretName: hc-secret
+      - name: fe-certs-vol
+        secret:
+          secretName: fe-secret
diff --git a/ingress/ingress-custom-grpc-health-check/example/fe-ilb-ingress.yaml b/ingress/ingress-custom-grpc-health-check/example/fe-ilb-ingress.yaml
@@ -0,0 +1,28 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: networking.k8s.io/v1beta1
+kind: Ingress
+metadata:
+  name: fe-ilb-ingress
+  annotations:
+    kubernetes.io/ingress.allow-http: "false"
+    kubernetes.io/ingress.class: "gce-internal"
+
+spec:
+  tls:
+  - secretName: fe-secret
+  backend:
+    serviceName: fe-srv-ingress
+    servicePort: 50051