Update Service with no change in Spec causes GCE to add/delete addresses and forwarding rules

[krmayankk]

I am updating K8s service on 1.1.3. There is no change in ServiceSpec, but on GCE console i see lots of logs related to addition and deletion of address and forwarding rules. Is this expected?

Specifically i see with one update of service(nothing changed in the spec):-
-create address
-delete forwarding rule
-create forwarding rule
-delete address

am i doing some mistake in updating a service. I do a getService to find the service and then mostly pass the same spec to UpdateService in the go client.
I also see that the Nodeport is always changing with each update of the service.

Is this expected, since i expect that if the spec remains the same, nothing should happe with Update service.
I can provide any other details needed to debug this?

[krmayankk]

Also i dont expect the resourceVersion to change, but the resourceVersion is also changing with each update, although i am not changing anything in the Spec.

[thockin]

Can you post the yaml or json you are using to POST and then using to PUT. Specifically, are you reading it back and then PUT or are you PUTting the original yaml twice?

[krmayankk]

i am using the unversioned go client.
-i do a getservice and save the resourceVersion
-when i find a service, i update its spec( the returnedvalue of getService) with new contents which in this case is not changing at all between updates and specify the saved resourceVersion.

I have pasted below the Get and PUT requests.

One thing i could be doing wrong is using the return value of GetService which is *api.Service in UpdateService, would that cause issue. I am creating LoadBalancer service.

I0328 14:30:54.099344 39264 round_trippers.go:267] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: sam-controller/v0.0.0 (darwin/amd64) kubernetes/$Format" http://localhost:9090/api/v1/namespaces/default/services/ma-shared-rptcoll1-0-phx
I0328 14:30:54.169726 39264 round_trippers.go:286] GET http://localhost:9090/api/v1/namespaces/default/services/ma-shared-rptcoll1-0-phx 200 OK in 70 milliseconds
I0328 14:30:54.169758 39264 round_trippers.go:292] Response Headers:
I0328 14:30:54.169766 39264 round_trippers.go:295] Content-Length: 643
I0328 14:30:54.169773 39264 round_trippers.go:295] Content-Type: application/json
I0328 14:30:54.169778 39264 round_trippers.go:295] Date: Mon, 28 Mar 2016 21:27:21 GMT
I0328 14:30:54.169831 39264 request.go:846] Response Body: {"kind":"Service","apiVersion":"v1","metadata":{"name":"ma-shared-rptcoll1-0-phx","namespace":"default","selfLink":"/api/v1/namespaces/default/services/ma-shared-rptcoll1-0-phx","uid":"cabfa600-f527-11e5-8cd9-42010af00002","resourceVersion":"2510991","creationTimestamp":"2016-03-28T20:58:15Z","labels":{"sam_loadbalancer":"shared-rptcoll1-0-phx","sam_ns":"ma"}},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":80,"nodePort":31476}],"selector":{"sam_function":"report-collector","sam_ns":"ma"},"clusterIP":"10.0.98.99","type":"LoadBalancer","sessionAffinity":"None"},"status":{"loadBalancer":{"ingress":[{"ip":"104.197.211.128"}]}}}
I0328 14:30:54.170073 39264 kubernetesclient.go:79] GetService Succeeded for ma-shared-rptcoll1-0-phx
I0328 14:30:54.170174 39264 request.go:551] Request Body: {"kind":"Service","apiVersion":"v1","metadata":{"name":"ma-shared-rptcoll1-0-phx","resourceVersion":"2510991","creationTimestamp":null,"labels":{"sam_loadbalancer":"shared-rptcoll1-0-phx","sam_ns":"ma"}},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":80}],"selector":{"sam_function":"report-collector","sam_ns":"ma"},"clusterIP":"10.0.98.99","type":"LoadBalancer","sessionAffinity":"None"},"status":{"loadBalancer":{"ingress":[{"ip":"104.197.211.128"}]}}}
I0328 14:30:54.170226 39264 round_trippers.go:267] curl -k -v -XPUT -H "Accept: application/json, /" -H "Content-Type: application/json" -H "User-Agent: sam-controller/v0.0.0 (darwin/amd64) kubernetes/$Format" http://localhost:9090/api/v1/namespaces/default/services/ma-shared-rptcoll1-0-phx
I0328 14:30:54.256569 39264 round_trippers.go:286] PUT http://localhost:9090/api/v1/namespaces/default/services/ma-shared-rptcoll1-0-phx 200 OK in 86 milliseconds
I0328 14:30:54.256601 39264 round_trippers.go:292] Response Headers:
I0328 14:30:54.256608 39264 round_trippers.go:295] Content-Type: application/json
I0328 14:30:54.256615 39264 round_trippers.go:295] Date: Mon, 28 Mar 2016 21:27:21 GMT
I0328 14:30:54.256621 39264 round_trippers.go:295] Content-Length: 643
I0328 14:30:54.256680 39264 request.go:846] Response Body: {"kind":"Service","apiVersion":"v1","metadata":{"name":"ma-shared-rptcoll1-0-phx","namespace":"default","selfLink":"/api/v1/namespaces/default/services/ma-shared-rptcoll1-0-phx","uid":"cabfa600-f527-11e5-8cd9-42010af00002","resourceVersion":"2511677","creationTimestamp":"2016-03-28T20:58:15Z","labels":{"sam_loadbalancer":"shared-rptcoll1-0-phx","sam_ns":"ma"}},"spec":{"ports":[{"protocol":"TCP","port":80,"targetPort":80,"nodePort":31705}],"selector":{"sam_function":"report-collector","sam_ns":"ma"},"clusterIP":"10.0.98.99","type":"LoadBalancer","sessionAffinity":"None"},"status":{"loadBalancer":{"ingress":[{"ip":"104.197.211.128"}]}}}
I0328 14:30:54.256929 39264 kubernetesclient.go:100] UpdateService Succeeded for ma-shared-rptcoll1-0-phx

[krmayankk]

I am trying to create an service of type LoadBalancer. If the service already exists and has already been allocated an external ip and nodeport, do i need to copy those from the getservice to updateService fields to preserve them and not lead to new allocations while Updating the Service.

I know how to preserve the NodePort already allocated.

I dont know how do that for external ip, do i need to preserve spec.LoadBalancerIP to Status.LoadBalancer.Ingress[0].IP for saving the external ip already allocated or do i need to copy the Ingress to ExternalIPs ? Its not very clear.

[thockin]

It's not supposed to change the external IP under any circumstances, and it doesn't in my own cluster.

Can you do the equivalent of this:

$ k get svc kubernetes -o yaml | grep -A5 status
status:
  loadBalancer: {}

$ k get svc kubernetes -o yaml | sed 's/ClusterIP/LoadBalancer/' | k replace -f -
service "kubernetes" replaced

$ while true; do if k get svc kubernetes -o yaml | grep ingress >/dev/null; then break; fi; done

$ k get svc kubernetes -o yaml | grep -A5 status
status:
  loadBalancer:
    ingress:
    - ip: 10.20.30.40

$ k annotate svc kubernetes foo=bar
service "kubernetes" annotated

$ sleep 300; k get svc kubernetes -o yaml | grep -A5 status
status:
  loadBalancer:
    ingress:
    - ip: 10.20.30.40

Or maybe you can post a minimal kubectl repro?

[krmayankk]

ok may be i am not correctly quantifying the problem here. My issues specifically are:-
1: if i call service Update(PUT) with no change in spec.Ports, type as loadbalancer, and all labels and selectors are also same, still i see the following
"loadBalancer": {
"ingress": [
{
"ip": "23.251.154.145"
}
]
}
portion of the loadbalancer, goes away(the service becomes unreachable for some time) and then is reallocated and on GCE i see the following sequence of calls again
-create address
-delete forwarding rule
-create forwarding rule
-delete address

in my experiment, i do see the external ip same , but my issue is when nothing has changed , why are we still calling GCE and reallocating the forwarding rule and address ?
The other issue is why are we allocating NodePort again and again when nothing has changed.

All this observation is on 1.1.3

[thockin]

Did you run the steps I ran above? Does it reproduce?

On Tue, Mar 29, 2016 at 10:41 PM, krmayankk notifications@github.com
wrote:

ok may be i am not correctly quantifying the problem here. My issues
specifically are:-
1: if i call service Update(PUT) with no change in spec.Ports, type as
loadbalancer, and all labels and selectors are also same, still i see the
following
"loadBalancer": {
"ingress": [
{
"ip": "23.251.154.145"
}
]
}
portion of the loadbalancer, goes away(the service becomes unreachable for
some time) and then is reallocated and on GCE i see the following sequence
of calls again
-create address
-delete forwarding rule
-create forwarding rule
-delete address

in my experiment, i do see the external ip same , but my issue is when
nothing has changed , why are we still calling GCE and reallocating the
forwarding rule and address ?
The other issue is why are we allocating NodePort again and again when
nothing has changed.

All this observation is on 1.1.3

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#23553 (comment)

[a-robinson]

You should be able to get a better sense of what's happening by looking at /var/log/kube-controller-manager.log on the master VM.

[krmayankk]

just moved to 1.2 now. So will need to verify if this is indeed a problem on 1.2 or not. Bear with me

[krmayankk]

@thockin it does not repro with the steps you posted above in 1.2. While we were relying on using the idempotent nature of the apis to cntinuosly call update and assume that if nothing changes in the app yaml, nothing should happen, we have shifted our model to diff the deployed/service spec with the spec being deployed and only if differs , we will call an update and also shifted to 1.2. So not sure if i can repro this anymore. All i was doing earlier is calling UpdateService from the go client again and again, while populating all the relevant fields in the spec, again and again as if its a new service, but preserving the resourceVersion from the getService call. At this point , i will either try to verify this behavior on 1.2 by changing code, which will probably have to wait until i have some free cycles.

[thockin]

Dup of #24180

[krmayankk]

Thanks for getting this fixed. For now in 1.2 i thought that was how you designed the nodepport feature and i had to preserve and merge the nodeports from old and new service updates to get this working. I am assuming that UpdateService after this fix, will preserve the existing nodeports even if the nodeports have not been explicitly assigned a port. Please confirm