Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UPSTREAM <carry>: Rebase code to kfp master branch - commit: 65d1d79fb7c77db076de0e9799516269c49e01a1 #134

Merged
merged 331 commits into from
Feb 12, 2025
Merged

UPSTREAM <carry>: Rebase code to kfp master branch - commit: 65d1d79fb7c77db076de0e9799516269c49e01a1 #134

merged 331 commits into from
Feb 12, 2025

Conversation

hbelmiro
Copy link

@hbelmiro hbelmiro commented Feb 5, 2025 •
edited
Loading

Description of your changes:
Rebase code to kfp master branch - commit: kubeflow@65d1d79

This PR should depends:

Checklist:

DharmitD and others added 30 commits July 26, 2024 20:42
@DharmitD
test: Migrate miscellaneous SDK tests to GHA Workflow (kubeflow#11032)
f1d5479 
* test: Migrate SDK isort tests to GHA

Signed-off-by: ddalvi <ddalvi@redhat.com>

* test: Migrate SDK docformatter tests to GHA

Signed-off-by: ddalvi <ddalvi@redhat.com>

* test: Migrate SDK Component YAML tests to GHA

Signed-off-by: ddalvi <ddalvi@redhat.com>

---------

Signed-off-by: ddalvi <ddalvi@redhat.com>
@gregsheremeta
chore: Adding @gregsheremeta to SDK reviewers (kubernetes_platform) (k…
97f2c66 
…ubeflow#11044)

Signed-off-by: Greg Sheremeta <gshereme@redhat.com>
@DharmitD
test: Migrate KFP SDK Tests to GHA (kubeflow#11037)
643c923 
Signed-off-by: ddalvi <ddalvi@redhat.com>
@hbelmiro
chore: adding @hbelmiro to kubernetes_platform/OWNERS (kubeflow#11045)
3324d67 
Signed-off-by: hbelmiro <helber.belmiro@gmail.com>
@hbelmiro
test: Moved kfp-kubernetes-execution-tests to GitHub Actions (kubeflo…
e940ba2 
…w#10926)

* Moved kfp-kubernetes-execution-tests to GitHub Actions

Signed-off-by: hbelmiro <helber.belmiro@gmail.com>

* Removed Python versions matrix

Signed-off-by: hbelmiro <helber.belmiro@gmail.com>

---------

Signed-off-by: hbelmiro <helber.belmiro@gmail.com>
@HumairAK
add dev docs for kfp/sdk (kubeflow#11046)
eea7d48 
Signed-off-by: Humair Khan <HumairAK@users.noreply.github.com>
feat(components): Support dynamic machine parameters in preview.custo…
e44dfa7 
…m_job.utils.create_custom_training_job_from_component

Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 657895789
@hbelmiro
test: Moved kubeflow-pipelines-samples-v2 to GitHub Actions (kubeflow…
87184fd 
…#11048)

Signed-off-by: hbelmiro <helber.belmiro@gmail.com>
feat(components): internal
461d892 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 658076101
feat(component): internal
ad03bb6 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 658108277
@diegolovison
test: Add script and GitHub Action for kfp-kubernetes-library (kubefl…
c420f8c 
…ow#11006)

* test: Add script and GitHub Action for kfp-kubernetes-library

Signed-off-by: Diego Lovison <diegolovison@gmail.com>

* Rename

Signed-off-by: Diego Lovison <diegolovison@gmail.com>

* Update bash to have sudo

Signed-off-by: Diego Lovison <diegolovison@gmail.com>

---------

Signed-off-by: Diego Lovison <diegolovison@gmail.com>
chore(components): GCPC 2.16.0 Release
ff4e918 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 658877559
@papagala
fix: Incorrect typing in samples/cores/loop_parallism and fixing loop…
1612dac 
…_parameter examples (kubeflow#11062)

* Fixing incorrect typing in loop_parallism example

Signed-off-by: Oswaldo Gomez <oswaldo.gomez@roche.com>

* Fixing samples/core/loop_parameter example

Signed-off-by: Oswaldo Gomez <oswaldo.gomez@roche.com>

---------

Signed-off-by: Oswaldo Gomez <oswaldo.gomez@roche.com>
Co-authored-by: Oswaldo Gomez <oswaldo.gomez@roche.com>
feat(components): internal
522b593 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 658993845
@hbelmiro
chore: Fixed e2e tests (kubeflow#11072)
6c4a250 
Signed-off-by: hbelmiro <helber.belmiro@gmail.com>
chore(components): Bump image version for Structured Data pipelines
65f6475 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 659655574
feat(components): internal
abe5257 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 659693795
feat(components): internal
e128bdb 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 659993682
feat(components): internal
c6f658b 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 660088225
@KevinGrantLee
feat(api): Add new ResourceSpec proto fields to support dynamic value…
83dcf1a 
…s. (kubeflow#11075)

Signed-off-by: KevinGrantLee <kglee@google.com>
feat(components): internal
2eb9bec 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 660148123
feat(components): internal
6f62203 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 660156029
@diegolovison
docs: Fix environment variables names (kubeflow#11013)
99bd234 
Signed-off-by: Diego Lovison <diegolovison@gmail.com>
@zazulam @droctothorpe
@andreafehrman @MonicaZhang1 @KyleKaminky @CarterFendley
feat(backend): move comp logic to workflow params (kubeflow#10979)
0e37fd6 
* feat(backend): move comp logic to workflow params

Signed-off-by: zazulam <m.zazula@gmail.com>
Co-authored-by: droctothorpe <mythicalsunlight@gmail.com>
Co-authored-by: andreafehrman <andrea.k.fehrman@vanderbilt.edu>
Co-authored-by: MonicaZhang1 <zhangmonica1@gmail.com>
Co-authored-by: kylekaminky <kyle.kaminky@gmail.com>
Co-authored-by: CarterFendley <carter.fendley@gmail.com>
Signed-off-by: zazulam <m.zazula@gmail.com>

* address pr comments

Signed-off-by: zazulam <m.zazula@gmail.com>

* Use function name instead of base name and address edge cases

Signed-off-by: droctothorpe <mythicalsunlight@gmail.com>
Co-authored-by: zazulam <m.zazula@gmail.com>

* Improve logic and update tests

Signed-off-by: droctothorpe <mythicalsunlight@gmail.com>
Co-authored-by: zazulam <m.zazula@gmail.com>

* POC hashing command and args

Signed-off-by: droctothorpe <mythicalsunlight@gmail.com>
Co-authored-by: zazulam <m.zazula@gmail.com>

* Add comments to clarify the logic

Signed-off-by: droctothorpe <mythicalsunlight@gmail.com>
Co-authored-by: zazulam <m.zazula@gmail.com>

* Hash entire PipelineContainerSpec

Signed-off-by: droctothorpe <mythicalsunlight@gmail.com>
Co-authored-by: zazulam <m.zazula@gmail.com>

---------

Signed-off-by: zazulam <m.zazula@gmail.com>
Signed-off-by: droctothorpe <mythicalsunlight@gmail.com>
Co-authored-by: droctothorpe <mythicalsunlight@gmail.com>
Co-authored-by: andreafehrman <andrea.k.fehrman@vanderbilt.edu>
Co-authored-by: MonicaZhang1 <zhangmonica1@gmail.com>
Co-authored-by: kylekaminky <kyle.kaminky@gmail.com>
Co-authored-by: CarterFendley <carter.fendley@gmail.com>
feat(component): internal
c06d9ae 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 660985413
feat(components): internal
e8f0208 
Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 661332120
fix(components): Fix to model batch explanation component for Structu…
289f64f 
…red Data pipelines

Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 661475667
feat(components): Support dynamic values for boot_disk_type, boot_dis…
7b7918e 
…k_size in preview.custom_job.utils.create_custom_training_job_from_component

Signed-off-by: Googler <nobody@google.com>
PiperOrigin-RevId: 662242688
@gmfrasca
chore: Upgrade Argo to v3.4.17 (kubeflow#10978)
3a68b60 
Signed-off-by: Giulio Frasca <gfrasca@redhat.com>
@VaniHaripriya
test: Moved kubeflow-pipelines-manifests to GitHub Actions (kubeflow#…
4458bdd 
…11066)

Signed-off-by: vmudadla <vmudadla@redhat.com>
@openshift-ci openshift-ci bot requested review from DharmitD and rimolive February 10, 2025 20:36
@hbelmiro
UPSTREAM <carry>: Removed non-RedHatters from OWNERS files
aee6811 
Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>
@dsp-developers
Copy link

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-134
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-134
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-134
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-134
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-134
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-134

mprahl and others added 3 commits February 10, 2025 22:10
@mprahl
feat(backend): Add support for job and task placeholders in the KFP b…
6a13f4b 
…ackend (kubeflow#11599)

This adds support for the following placeholders in the KFP backend:

- dsl.PIPELINE_JOB_NAME_PLACEHOLDER
- dsl.PIPELINE_JOB_RESOURCE_NAME_PLACEHOLDER
- dsl.PIPELINE_JOB_ID_PLACEHOLDER
- dsl.PIPELINE_TASK_NAME_PLACEHOLDER
- dsl.PIPELINE_TASK_ID_PLACEHOLDER

Resolves:
kubeflow#10453

Signed-off-by: mprahl <mprahl@users.noreply.github.com>
@dandawg
chore(backend): updated deprecated grpc func (kubeflow#11603)
65d1d79 
Signed-off-by: Daniel Dowler <12484302+dandawg@users.noreply.github.com>
@hbelmiro
Merge remote-tracking branch 'kubeflow/master' into rebase-2.4-master
94184c6 
# Conflicts:
#	backend/src/v2/cacheutils/cache.go
#	backend/src/v2/cmd/driver/main.go
#	backend/src/v2/driver/driver.go
@hbelmiro hbelmiro changed the title UPSTREAM <carry>: Rebase code to kfp master branch - commit: c100648fa8c36995eb6236ef402c37763d935da8 UPSTREAM <carry>: Rebase code to kfp master branch - commit: 65d1d79fb7c77db076de0e9799516269c49e01a1 Feb 11, 2025
@hbelmiro
Copy link
Author

Added new commits from upstream.

@dsp-developers
Copy link

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-134
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-134
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-134
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-134
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-134
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-134

@mprahl
Copy link

mprahl commented Feb 11, 2025

/cc @mprahl

@openshift-ci openshift-ci bot requested a review from mprahl February 11, 2025 14:46
@hbelmiro
Merge remote-tracking branch 'opendatahub-io/master' into rebase-2.4-…
cc29292 
…master

# Conflicts:
#	backend/third_party_licenses/apiserver.csv
#	backend/third_party_licenses/cache_server.csv
#	backend/third_party_licenses/driver.csv
#	backend/third_party_licenses/launcher.csv
#	backend/third_party_licenses/persistence_agent.csv
#	backend/third_party_licenses/swf.csv
#	backend/third_party_licenses/viewer.csv
#	go.mod
#	go.sum
@dsp-developers
Copy link

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-134
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-134
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-134
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-134
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-134
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-134

@rimolive
Copy link

Code LGTM, and I tested the images build by dsp-developers bot in a testing environment. iris-pipeline example ran fine.

However, in my testing environment, the scheduled-workflow controller pod is always restarting with the following message:

time="2025-02-12T15:46:51Z" level=info msg="Connecting the API server over GRPC at: ml-pipeline:8887"
time="2025-02-12T15:47:51Z" level=fatal msg="Error connecting to the API server after trying for one minute: Failed to create gRPC connection: context deadline exceeded"

This was caused by this commit that introduced new parameters to specify the API Server hostname/port. I don't think that blocks merging this PR, but we must ensure we have this change covered in DSPO repo.

cc @mprahl @hbelmiro

@hbelmiro
Copy link
Author

However, in my testing environment, the scheduled-workflow controller pod is always restarting with the following message:

time="2025-02-12T15:46:51Z" level=info msg="Connecting the API server over GRPC at: ml-pipeline:8887"
time="2025-02-12T15:47:51Z" level=fatal msg="Error connecting to the API server after trying for one minute: Failed to create gRPC connection: context deadline exceeded"

This was caused by this commit that introduced new parameters to specify the API Server hostname/port. I don't think that blocks merging this PR, but we must ensure we have this change covered in DSPO repo.

@rimolive you might have forgotten to use opendatahub-io/data-science-pipelines-operator#779.

@rimolive
Copy link

/lgtm

Thanks, @hbelmiro! Using the image build in opendatahub-io/data-science-pipelines-operator#779 fixed the ScheduledWorkflow controller issue.

@openshift-ci openshift-ci bot added the lgtm label Feb 12, 2025
mprahl
mprahl approved these changes Feb 12, 2025
View reviewed changes
Copy link

@mprahl mprahl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a git diff comparing your PR to the latest upstream master and it looked good to me.

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Feb 12, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mprahl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit d347d3e into opendatahub-io:master Feb 12, 2025
60 checks passed
@hbelmiro hbelmiro deleted the rebase-2.4-master branch February 13, 2025 11:49
@mprahl mprahl mentioned this pull request Feb 13, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mprahl mprahl mprahl approved these changes

@DharmitD DharmitD Awaiting requested review from DharmitD

@rimolive rimolive Awaiting requested review from rimolive

Assignees

@rimolive rimolive

@mprahl mprahl

Labels
approved lgtm
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.