Avoid depending on port-forwarded ambassador service

[IronPan]

Kubeflow is moving away from ambassador, in favor of istio, and proxy the cluster using cloud endpoint with IAP or simple auth. In both cases, we should use the ingress endpoint instead of depending on port-forwarding to access the cluster.

[kevinbache]

somewhat related: #1104, #780

[jlewi]

@IronPan @vicaire How's this coming? Are you on track to have this ready for 0.6?

[IronPan]

@jingzhang36 I remember you are working on istio integration. How is it going?
@jlewi Just to double check. I think we agreed that despite istio integration, we always leave user an option to not install istio, and can do port-forwarding. Is that something that still align with your thinking?

[jlewi]

@IronPan I think we discussed this offline. ISTIO or a service mesh is a requirement. If users want to customize the ISTIO gateway config to disable auth to allow port-forwarding they can do that. This would require editing various config files to disable AuthZ at the ISTIO gateway.

[jlewi]

@IronPan Is this done?

[IronPan]

@jingzhang36 Do you have any update on this?
@hongye-sun I think we should do an evaluation on the kfp.client() when the 0.6 candidate is out. WDYT?
Let's not close this before both items above are done.

[jlewi]

@IronPan @hongye-sun I would suggest trying out master as soon as possible to see if its working.

[jlewi]

IAP should be working on master; basic auth seems to be having problems on master #3562

[jingzhang36]

Regarding istio, I feel tensorboard /data hack is no longer blocking istio since #1237 is recently merged (and Gabriel's PR kubeflow/kubeflow#3036 is already in master). Are there any other things that need to be done to complete istio integration? E.g., remove old routing rules?

[jlewi]

@IronPan Have you or someone else verified that the pipelines SDK works with IAP? I believe that was the original genesis of this issue. In particular the goal of this issue was to ensure the could connect via the secure public endpoint of KF to pipelines from outside the cluster.

/cc @jessiezcc

[IronPan]

Verified the KFP client works well with a v0.6 KF cluster with IAP.
Here is a code snippet that can successfully create/list experiment

import kfp
client=kfp.Client(namespace="kubeflow")
exp=client.create_experiment(name="commandline")
client.list_experiments()