build_image_from_working_dir fails with AccessDenied Error

[jonasboecquaert]

What steps did you take:

I wanted to create a custom image to use in my pipeline. The code i executed is the following:

I'm however getting an error when i try executing this code block.

What happened:

I got an AccessDenied error

What did you expect to happen:

I expected an image to be build and pushed on google container registry.

Environment:

I deployed KFP with CLI, using the documentation found here

KFP version: Build commit: ca58b22

KFP SDK version: 0.2.5

Anything else you would like to add:

I'm trying to run the code in a jupyter notebook on a jupyter server running on my KFP deployment.
I found that the folder kfp_container_build_staging in my storage bucket contains the file 8e5ca5cd-fb34-47e1-a496-aebe02fb3a83.tar.gz. So the problem isn't in the building of the image.
A colleague could run the same codeblock without any problems on an earlier version of KFP. I did some research as why this might be happening. In my kaniko-workload pod i found that the user-gcp-sa secret isn't included in the YAML. So I went through the code and found that the container builder code was changed to not use that gcp secret anymore here: #2619.

[jonasboecquaert]

@Svendegroote91

[rmgogogo]

Would you help detail which mode the cluster is on? is it workload_identity enabled or created with "--scopes cloud_platform"?
More info: https://www.kubeflow.org/docs/pipelines/installation/standalone-deployment/#set-up-your-cluster

If it's workload identity, please make sure you did the Google Service Account binding with K8s Service Account.

From the error log, it said no permission to access storage bucket. Please check the bucket permission to see if the Google Service Account is able to access it.

[jonasboecquaert]

unfortunately I can't provide that information anymore, as I removed the cluster in favor of using the managed AI platform pipelines.

I don't think I have set those scopes or enabled workload_identity, so this could very well have been the problem.

[yantriks-edi-bice]

I upgraded pipelines to 0.4 on Kubeflow 1.0.2 on GCP

I realize the fix to specify service account was merged post 0.4 release so I'm still patching manually

!sed -i "s/'serviceAccountName': 'default'/'serviceAccountName': 'default-editor'/" /home/jovyan/.local/lib/python3.6/site-packages/kfp/containers/_container_builder.py

Though the following continues to fail with "

Primary: /namespaces/saas-ml-dev.svc.id.goog with additional claims does not have storage.objects.get access to kf-poc-edi/markdowns-tfx-pipeline/kfp_container_build_staging/cc7c17a5-45d5-49b0-bc7e-e61c09f5e16c.tar.gz.

"

import kfp
b = ContainerBuilder(gcs_staging=s, namespace='kubeflow-edi-bice')
kfp.containers.build_image_from_working_dir(builder=b)

Even though I do have the RBAC mapping and the IAM permissions granting kf-tst-user StorageAdmin role on that bucket

kubectl -n kubeflow-edi-bice describe serviceaccount default-editor
Name: default-editor
Namespace: kubeflow-edi-bice
Labels:
Annotations: iam.gke.io/gcp-service-account: kf-tst-user@saas-ml-dev.iam.gserviceaccount.com
Image pull secrets:
Mountable secrets: kfserving-secret-j58kx
kfserving-secret-5xrd9
kfserving-secret-qhk2k
kfserving-secret-gjsmc
kfserving-secret-f9ljp
default-editor-token-rmjlr
Tokens: default-editor-token-rmjlr
Events:

[yantriks-edi-bice]

I am getting further along - past this issue - after deleting all metadata deployments (may have broken things following the pipelines upgrade instructions which apparently were for standalone pipelines) and reapplying kfctl.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.