resource/kubernetes_service: Wait for LoadBalancer ingress

[radeksimko]

This is to allow the user to easily access the exposed IP/hostname for the app, e.g.

output "lb_ip" {
  value = "${kubernetes_service.example.load_balancer_ingress.0.ip}"
}

[e-max]

Awesome! I've started to write similar PR, but yours much better. How do you run acceptance tests? I've tried to run it against GKE cluster, but it seems that it doesn't support the test when you are explicitly set LoadBalancerIP.

[radeksimko]

Ha, good question - I do have a configuration of service w/ type = "LoadBalancer" deployed onto GKE cluster and that works just fine. I should probably add that somewhere to the test suite. Thanks for the ping.

I never tried to set the load_balancer_ip/LoadBalancerIP explicitly. I don't think it's supported by GKE. I do have long-term plans for running all acceptance tests in other environments, but atm it's only being tested nightly in GKE.

[e-max]

Hm. I probably misunderstood something. I am looking at resource_kubernetes_service_test.go and it look like the spec for this test is

        external_name = "ext-name-%s"
        external_ips = ["10.0.0.3", "10.0.0.4"]
        load_balancer_ip = "12.0.0.120"
        load_balancer_source_ranges = ["10.0.0.5/32", "10.0.0.6/32"]
        selector {
            App = "MyApp"
        }
        session_affinity = "ClientIP"
        port {
            port = 8888
            target_port = 80
        }
        type = "LoadBalancer"

When I run this test against GKE cluster I can see something like this

Name:                   tf-acc-test-wl8i60itlw
Namespace:              default
Labels:                 <none>
Annotations:            <none>
Selector:               App=MyApp
Type:                   LoadBalancer
IP:                     10.7.248.112
External IPs:           10.0.0.4,10.0.0.3
External Name:          ext-name-tf-acc-test-wl8i60itlw
Port:                   <unset> 8888/TCP
NodePort:               <unset> 31838/TCP
Endpoints:              <none>
Session Affinity:       ClientIP
Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath   Type            Reason                          Message
  ---------     --------        -----   ----                    -------------   --------        ------                          -------
  3h            1m              43      service-controller                      Normal          CreatingLoadBalancer            Creating load balancer
  3h            1m              43      service-controller                      Warning         CreatingLoadBalancerFailed      Error creating load balancer (will retry): Failed to create load balancer for service default/tf-acc-test-wl8i60itlw: requested ip 12.0.0.120 is neither static nor assigned to LB a21ff797757fb11e7b16442010a840fd(default/tf-acc-test-wl8i60itlw): <nil>

[radeksimko]

You're absolutely right that the config we have in the acceptance test is creating service which doesn't actually work in GKE because it cannot assign the IP and this PR is exposing that fact by making the test fail.

I'll rework the test + see if I can pull the failures from the events API and show the reason to the user.

[radeksimko]

Failing test fixed, thanks again for noticing.

make testacc TEST=./kubernetes TESTARGS='-run=TestAccKubernetesService_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./kubernetes -v -run=TestAccKubernetesService_ -timeout 120m
=== RUN   TestAccKubernetesService_basic
--- PASS: TestAccKubernetesService_basic (5.26s)
=== RUN   TestAccKubernetesService_loadBalancer
--- PASS: TestAccKubernetesService_loadBalancer (52.47s)
=== RUN   TestAccKubernetesService_nodePort
--- PASS: TestAccKubernetesService_nodePort (2.25s)
=== RUN   TestAccKubernetesService_importBasic
--- PASS: TestAccKubernetesService_importBasic (2.75s)
=== RUN   TestAccKubernetesService_generatedName
--- PASS: TestAccKubernetesService_generatedName (2.44s)
=== RUN   TestAccKubernetesService_importGeneratedName
--- PASS: TestAccKubernetesService_importGeneratedName (2.14s)
PASS
ok  	github.com/terraform-providers/terraform-provider-kubernetes/kubernetes	67.390s

I will address the exposure of failure reasons in a separate PR as it doesn't affect just k8s service, but also Pod, RC and PVC.

[e-max]

Awesome! Thank you!

By the way - could you check if terraform destroy correctly removes LoadBalancer?
I have a terraform file which creates GKE cluster and creates service with type=LoadBalancer. . Everything works fine but when I call terraform destroy it deletes the cluster, but LoadBalancer left alive.

[radeksimko]

@e-max The deletion of LBs and associated resources is AFAIK asynchronous and I don't think we have access to the status of that operation through K8S API. I can tell it always gets deleted successfully, within ~20-30secs after deletion of the service. Below is relevant snippet from the event log generated by a terraform destroy on a single kubernetes_service resource:

gcloud compute forwarding-rules list also reports the FWD rule is missing after the mentioned time.

[tombuildsstuff]

a couple of minor questions - otherwise LGTM :)

[tombuildsstuff]

do we want to check the status code before acting, or will the Get method above handle returning an error for a non-ok status code?

[radeksimko]

As discussed via Slack - in case the service suddenly doesn't exist it would get caught by the err check above, if the creation of LB failed then we'd time out after 5 mins. Exposing the reason of failed creation in the timeout error message is a job for a separate PR.

[tombuildsstuff]

should we add a test to check this value?

[radeksimko]

The K8S always returned empty struct consistently - i.e. svc.Status was never nil so I don't think so.