Make user-gcp-sa secret optional on GCP

[Bobgy]

UPDATE:
When GKE provides application default credentials with enough permission through one of the following three ways:

• Project default service account
• User specified GCE default service account (cluster or node pool level fine grained control)
• Workload identity (kubernetes service account level fine grained control)

There's no need to set user-gcp-sa in https://github.com/kubeflow/pipelines/blob/master/manifests/gcp_marketplace/guide.md#gcp-service-account-credentials. This allows zero config on-boarding experience.

This issue tracks various efforts towards making user-gcp-sa optional.

Pipeline samples:

• If user-gcp-sa is not present, drop GOOGLE_APPLICATION_CREDENTIALS env in pipelines api server before running it. Drop GCP credentials env if user-gcp-sa secret is not present #2643
  With the above done, we no longer need to change all pipeline samples with GCP auth usage This PR is abandoned to avoid making short-term workarounds that we need to remove in the future too.

Hidden links to samples

* [ ] https://github.com/kubeflow/pipelines/blob/d5e27e291fbe09cfa63a9de0f5b4fdea65af7c6f/sdk/python/kfp/gcp.py#L18
* [ ] https://github.com/kubeflow/pipelines/tree/master/samples/core/xgboost_training_cm
* [ ] https://github.com/kubeflow/pipelines/tree/master/samples/contrib/parameterized_tfx_oss
* [ ] [tfx sample](https://github.com/tensorflow/tfx/blob/6b6a2f9b69c93ce8a889c48d9e02354ed52c3f89/tfx/orchestration/kubeflow/kubeflow_dag_runner.py#L109)

Servers

• frontend node server should not use user-gcp-sa on Kubeflow deployment
• CloudSqlProxy /credentials/gcp-sa-token on marketplace
• Minio /etc/credentials/gcp-sa-token on marketplace
• TensorBoardViewer

[Bobgy]

/cc @rmgogogo

[Ark-kun]

The SDK extension method is probably still useful, but I agree that the samples should be cleaned.
We need to verify that everything works out of the box on all deployments:

• Marketplace
• Standalone
• Kubeflow

[IronPan]

Using default SA is not considered a good practice https://cloud.google.com/kubernetes-engine/docs/tutorials/authenticating-to-cloud-platform#why_use_service_accounts

• No visibility on how it's setup
• Harder to revoke permission
• It leads to a misleading habit for user to setup and will be a debt to remove when supporting workload identity.

[Bobgy]

Efforts are paused because we agreed that KFP will eventually move all deployments to workload identity. When that happens, we have no need for taking credentials.

On-prem will still need this, but it's not our priority. We can revisit the issue later.

[Bobgy]

Updated description to better reflect current status. Bumped to p0 because it blocks both multi user efforts and marketplace efforts.

[Bobgy]

Migrate standalone deployment to workload identity on GCP #2619 removes all use_gcp_secret usages in samples, so they can be run in both workload identity and GCP hosted ML pipelines

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bobgy]

We are generally recommending workload identity now because it reached GA. Closing the issue.