Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(backend): Implement aws-go-sdk crendentials to support IRSA for s3 in V2 #9502

Merged
merged 2 commits into from
May 26, 2023

Conversation

ryansteakley
Copy link
Contributor

@ryansteakley ryansteakley commented May 24, 2023

Description of your changes:
Use the aws-go-sdk credentials.ChainProvider to support IRSA in kfp v2. This functionality is already supported in V1, this pr aims to keep parity with v1 as IRSA is supported in v1 backend

func createCredentialProvidersChain(endpoint, accessKey, secretKey string) *credentials.Credentials {
Similar work was done for V1 frontend here #8651. The goal is to maintain IRSA support in Kubeflow Pipelines as V2 goes GA following the effort to support S3 Integration for Artifact Storage (IRSA) in this issue #8502.

Checklist:

@ryansteakley
Copy link
Contributor Author

ryansteakley commented May 24, 2023

@zijianjoy @chensun

@surajkota
Copy link
Contributor

@ryansteakley link the previous PRs and issues for background

@zijianjoy
Copy link
Collaborator

/assign @chensun

@chensun
Copy link
Member

chensun commented May 25, 2023

/retest

Copy link
Member

@chensun chensun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@@ -53,10 +53,10 @@ func OpenBucket(ctx context.Context, k8sClient kubernetes.Interface, namespace s
if config.Scheme == "minio://" {
cred, err := getMinioCredential(ctx, k8sClient, namespace)
if err != nil {
return nil, fmt.Errorf("Failed to get minio credential: %w", err)
return nil, fmt.Errorf("Failed to get minio credential: %w", err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove trailing whitespace

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@google-oss-prow google-oss-prow bot added the lgtm label May 26, 2023
Copy link
Member

@chensun chensun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@google-oss-prow google-oss-prow bot removed the lgtm label May 26, 2023
@ryansteakley
Copy link
Contributor Author

@chensun have fixed the whitespace

Copy link
Member

@chensun chensun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

Thanks!

@google-oss-prow google-oss-prow bot added the lgtm label May 26, 2023
@google-oss-prow
Copy link

@ryansteakley: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
kubeflow-pipelines-samples-v2 0c5e662 link false /test kubeflow-pipelines-samples-v2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit 8a3215e into kubeflow:master May 26, 2023
}

func getAWSCredential() (cred *credentials.Credentials, err error) {
return credentials.NewCredentials(&credentials.ChainProvider{}), nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ryansteakley are you sure that you did not forget to specify the providers in the ChainProvider?

Because by default a chain provider has no internal providers and will always error.

@thesuperzapper
Copy link
Member

@ryansteakley @chensun I am pretty sure this code will never get called, because any s3 bucket path would start with s3:// as the getMinioCredential() method is only run if the path starts with minio://, see here:

However, is is possible that IRSA was working before this PR if blob.OpenBucket correctly gets the auth from the environment variables when it sees an s3:// bucket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants