[gcp] Enable new stackdriver Kubernetes agents on GKE

[jlewi]

To enable the new agents I think we need to set the following in our cluster DM config.

loggingService: logging.googleapis.com/kubernetes
monitoringService: monitoring.googleapis.com/kubernetes

We can verify its enabled in the Cloud console where it should say

Stackdriver Logging	
Enabled v2(beta)
Stackdriver Monitoring	
Enabled v2(beta)

[jlewi]

I think this might need a little bit more work.

It looks like its guarded by gkeApiVersion

kubeflow/deployment/gke/deployment_manager_configs/cluster.jinja

Line 76 in 2e03ae9

{% if properties['gkeApiVersion'] == 'v1beta1' %}

We just need to test it and then maybe document it.

[chrisheecho]

/remove-priority p1

[jlewi]

/priority p1

[jlewi]

It looks like the feature is still in beta.
https://cloud.google.com/monitoring/kubernetes-engine/

Beta features are only available through the v1beta1 API
https://cloud.google.com/kubernetes-engine/docs/reference/api-organization#beta

So we need to use that API to create/modify the cluster.

[jlewi]

Confirmed on https://kf-v0-4-n00.endpoints.kubeflow-ci.cloud.goog/ that new stackdriver agents are deployed.

that we are now enabling the new stackedriver agents and that

Using the query

resource.type="k8s_container"
resource.labels.cluster_name="kf-v0-4-n00"
resource.labels.pod_name="ambassador-5cf8cd97d5-7xxdv"

Response

{
 insertId:  "1lfq8cxfvwlpb9"  
 logName:  "projects/kubeflow-ci/logs/stdout"  
 metadata: {
  systemLabels: {…}   
  userLabels: {
   pod-template-hash:  "1794785381"    
   service:  "ambassador"    
  }
 }
 receiveTimestamp:  "2019-01-06T20:08:49.315277338Z"  
 resource: {…}  
 severity:  "INFO"  
 textPayload:  "ACCESS [2019-01-06T20:08:37.965Z] "GET /argo/api/workflows/live HTTP/1.1" 504 UT 0 24 3000 - "108.222.64.88, 35.244.138.24" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36" "cc07b564-3e19-424e-83d2-4304f2166b6f" "kf-v0-4-n00.endpoints.kubeflow-ci.cloud.goog" "10.43.254.44:80"
"  
 timestamp:  "2019-01-06T20:08:47Z"  
}