UPSTREAM:<carry>:fix:Endpoint fix for AWS S3 Bucket Session #30

amadhusu · 2024-03-21T14:25:55Z

Description of your changes:
Resolves RHOAIENG-3774 . This resolves the issue of having to specify the region within the host field in DSPA after already providing the same in "Region" field.

Testing Instructions:

Deploy DSPO
Create the secret 'aws-bucket-creds' in the namespace where you are going to deploy the DSPA.
Deploy DSPA [a] attached below where the host is 's3.amazonaws.com'
Run the sample Pipeline and ensure it executes properly.
Deploy DSPA [b] attached below where the host is 's3.us-east-2.amazonaws.com'
Run the sample Pipeline and ensure it executes properly again.

DSPA [a]

apiVersion: datasciencepipelinesapplications.opendatahub.io/v1alpha1
kind: DataSciencePipelinesApplication
metadata:
  name: pr-30
spec:
  dspVersion: v2
  apiServer:
    enableSamplePipeline: true
    image: "quay.io/opendatahub/ds-pipelines-api-server:pr-30"
    argoDriverImage: "quay.io/opendatahub/ds-pipelines-driver:pr-30"
    argoLauncherImage: "quay.io/opendatahub/ds-pipelines-launcher:pr-30"
  persistenceAgent:
    image: "quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30"
  scheduledWorkflow:
    image: "quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30"
  mlmd:  
    deploy: true  # Optional component
    grpc:
      image: "quay.io/opendatahub/mlmd-grpc-server:latest"
    envoy:
      image: "registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2"
  mlpipelineUI:
    deploy: true  # Optional component 
    image: "quay.io/opendatahub/ds-pipelines-frontend:pr-30"
  objectStorage:
    externalStorage:
      bucket: rhods-dsp-dev
      host: s3.amazonaws.com
      region: us-east-2
      s3CredentialsSecret:
        accessKey: k8saccesskey
        secretKey: k8ssecretkey
        secretName: aws-bucket-creds
      scheme: https

DSPA [b]

apiVersion: datasciencepipelinesapplications.opendatahub.io/v1alpha1
kind: DataSciencePipelinesApplication
metadata:
  name: pr-30
spec:
  dspVersion: v2
  apiServer:
    enableSamplePipeline: true
    image: "quay.io/opendatahub/ds-pipelines-api-server:pr-30"
    argoDriverImage: "quay.io/opendatahub/ds-pipelines-driver:pr-30"
    argoLauncherImage: "quay.io/opendatahub/ds-pipelines-launcher:pr-30"
  persistenceAgent:
    image: "quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30"
  scheduledWorkflow:
    image: "quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30"
  mlmd:  
    deploy: true  # Optional component
    grpc:
      image: "quay.io/opendatahub/mlmd-grpc-server:latest"
    envoy:
      image: "registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2"
  mlpipelineUI:
    deploy: true  # Optional component 
    image: "quay.io/opendatahub/ds-pipelines-frontend:pr-30"
  objectStorage:
    externalStorage:
      bucket: rhods-dsp-dev
      host: s3.us-east-2.amazonaws.com
      region: us-east-2
      s3CredentialsSecret:
        accessKey: k8saccesskey
        secretKey: k8ssecretkey
        secretName: aws-bucket-creds
      scheme: https

Checklist:

The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

dsp-developers · 2024-03-21T14:34:52Z

A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-30
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-30
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-30
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-30

dsp-developers · 2024-03-21T14:34:53Z

An OCP cluster where you are logged in as cluster admin is required.

The Data Science Pipelines team recommends testing this using the Data Science Pipelines Operator. Check here for more information on using the DSPO.

To use and deploy a DSP stack with these images (assuming the DSPO is deployed), first save the following YAML to a file named dspa.pr-30.yaml:

apiVersion: datasciencepipelinesapplications.opendatahub.io/v1alpha1
kind: DataSciencePipelinesApplication
metadata:
  name: pr-30
spec:
  dspVersion: v2
  apiServer:
    image: "quay.io/opendatahub/ds-pipelines-api-server:pr-30"
    argoDriverImage: "quay.io/opendatahub/ds-pipelines-driver:pr-30"
    argoLauncherImage: "quay.io/opendatahub/ds-pipelines-launcher:pr-30"
  persistenceAgent:
    image: "quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30"
  scheduledWorkflow:
    image: "quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30"
  mlmd:  
    deploy: true  # Optional component
    grpc:
      image: "quay.io/opendatahub/mlmd-grpc-server:latest"
    envoy:
      image: "registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2"
  mlpipelineUI:
    deploy: true  # Optional component 
    image: "quay.io/opendatahub/ds-pipelines-frontend:pr-30"
  objectStorage:
    minio:
      deploy: true
      image: 'quay.io/opendatahub/minio:RELEASE.2019-08-14T20-37-41Z-license-compliance'

Then run the following:

cd $(mktemp -d)
git clone git@github.com:opendatahub-io/data-science-pipelines.git
cd data-science-pipelines/
git fetch origin pull/30/head
git checkout -b pullrequest adeec9b36f0948ef03a057dd59fa234234427633
oc apply -f dspa.pr-30.yaml

More instructions here on how to deploy and test a Data Science Pipelines Application.

dsp-developers · 2024-04-08T15:59:57Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-30
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-30
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-30
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-30

dsp-developers · 2024-04-09T10:42:42Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-30
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-30
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-30
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-30

dsp-developers · 2024-04-09T13:47:08Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-30
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-30
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-30
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-30

dsp-developers · 2024-04-09T14:32:49Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-30
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-30
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-30
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-30

dsp-developers · 2024-04-09T15:27:20Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-30
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-30
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-30
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-30

dsp-developers · 2024-04-09T16:09:27Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-30
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-30
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-30
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-30

dsp-developers · 2024-04-09T19:30:01Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-30
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-30
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-30
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-30

dsp-developers · 2024-04-11T18:25:15Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-30
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-30
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-30
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-30

backend/src/v2/objectstore/object_store.go

Signed-off-by: Achyut Madhusudan <amadhusu@redhat.com>

dsp-developers · 2024-04-16T12:27:54Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-30
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-30
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-30
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-30
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-30
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-30

HumairAK · 2024-04-16T19:07:40Z

deployed, both versions of the dspa's provided, api server strated both when region is specified in endpoint, and when it's not

/lgtm
/approve

openshift-ci · 2024-04-16T19:07:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: HumairAK

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [HumairAK]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot added the do-not-merge/work-in-progress label Mar 21, 2024

openshift-ci bot requested review from DharmitD and VaniHaripriya March 21, 2024 14:26

amadhusu force-pushed the RHOAIENG-3774 branch from adeec9b to 1b550d6 Compare April 8, 2024 15:50

amadhusu changed the title ~~WIP:UPSTREAM:<carry>:fix:Updated the aws-sdk-go version~~ WIP:UPSTREAM:<carry>:fix:Endpoint fix for AWS S3 Bucket Session Apr 8, 2024

amadhusu force-pushed the RHOAIENG-3774 branch from 1b550d6 to a8b1167 Compare April 9, 2024 10:33

amadhusu force-pushed the RHOAIENG-3774 branch from 7428dc7 to 31d9afc Compare April 9, 2024 15:18

amadhusu force-pushed the RHOAIENG-3774 branch from 31d9afc to 1356026 Compare April 9, 2024 16:00

amadhusu force-pushed the RHOAIENG-3774 branch from 1356026 to 8875e3e Compare April 9, 2024 19:21

amadhusu changed the title ~~WIP:UPSTREAM:<carry>:fix:Endpoint fix for AWS S3 Bucket Session~~ UPSTREAM:<carry>:fix:Endpoint fix for AWS S3 Bucket Session Apr 9, 2024

openshift-ci bot removed the do-not-merge/work-in-progress label Apr 9, 2024

HumairAK reviewed Apr 15, 2024

View reviewed changes

backend/src/v2/objectstore/object_store.go Outdated Show resolved Hide resolved

HumairAK reviewed Apr 15, 2024

View reviewed changes

backend/src/v2/objectstore/object_store.go Outdated Show resolved Hide resolved

HumairAK reviewed Apr 15, 2024

View reviewed changes

backend/src/v2/objectstore/object_store.go Outdated Show resolved Hide resolved

UPSTREAM:<carry>:fix:Endpoint fix for AWS S3 Bucket Session

502e439

Signed-off-by: Achyut Madhusudan <amadhusu@redhat.com>

amadhusu force-pushed the RHOAIENG-3774 branch from 8875e3e to 502e439 Compare April 16, 2024 12:18

amadhusu requested a review from HumairAK April 16, 2024 18:42

openshift-ci bot assigned HumairAK Apr 16, 2024

openshift-ci bot added the lgtm label Apr 16, 2024

openshift-ci bot added the approved label Apr 16, 2024

openshift-merge-bot bot merged commit 95cfc15 into opendatahub-io:master Apr 16, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UPSTREAM:<carry>:fix:Endpoint fix for AWS S3 Bucket Session #30

UPSTREAM:<carry>:fix:Endpoint fix for AWS S3 Bucket Session #30

amadhusu commented Mar 21, 2024 •

edited

Loading

dsp-developers commented Mar 21, 2024

dsp-developers commented Mar 21, 2024

dsp-developers commented Apr 8, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 11, 2024

dsp-developers commented Apr 16, 2024

HumairAK commented Apr 16, 2024 •

edited

Loading

openshift-ci bot commented Apr 16, 2024

UPSTREAM:<carry>:fix:Endpoint fix for AWS S3 Bucket Session #30

UPSTREAM:<carry>:fix:Endpoint fix for AWS S3 Bucket Session #30

Conversation

amadhusu commented Mar 21, 2024 • edited Loading

dsp-developers commented Mar 21, 2024

dsp-developers commented Mar 21, 2024

dsp-developers commented Apr 8, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 9, 2024

dsp-developers commented Apr 11, 2024

dsp-developers commented Apr 16, 2024

HumairAK commented Apr 16, 2024 • edited Loading

openshift-ci bot commented Apr 16, 2024

amadhusu commented Mar 21, 2024 •

edited

Loading

HumairAK commented Apr 16, 2024 •

edited

Loading