feat(sdk): Support dsl.ParallelFor over list of Artifacts #10441

KevinGrantLee · 2024-01-29T21:55:10Z

Description of your changes:

This PR adds support over dsl.ParallelFor over tasks that output lists of artifacts.

ex.

@dsl.component
def make_artifacts(...) -> List[Artifact]
    ...

@dsl.pipeline
def my_pipeline:
    make_artifacts_task = make_artifacts()
    with dsl.ParallelFor(items=make_artifacts_task.output) as item:
        print_artifact_name(var_artifact=item)

This PR does not support dsl.ParallelFor over a raw list of Artifacts.

Checklist:

The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

sdk/RELEASE.md

sdk/python/kfp/compiler/compiler_test.py

sdk/python/kfp/dsl/tasks_group.py

sdk/python/kfp/dsl/pipeline_channel.py

sdk/python/kfp/compiler/pipeline_spec_builder.py

sdk/python/kfp/dsl/for_loop.py

…nto parallelfor-list-artifacts

…GrantLee/pipelines into parallelfor-list-artifacts

sdk/python/kfp/dsl/for_loop.py

sdk/python/kfp/dsl/pipeline_channel.py

sdk/python/kfp/dsl/tasks_group.py

sdk/python/kfp/compiler/pipeline_spec_builder.py

sdk/python/kfp/dsl/types/type_annotations.py

sdk/python/kfp/dsl/for_loop.py

changes maded

sdk/python/kfp/dsl/for_loop.py

sdk/python/kfp/dsl/for_loop_test.py

google-oss-prow · 2024-02-06T22:38:52Z

@KevinGrantLee: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
kfp-kubernetes-execution-tests	`d367372`	link	false	`/test kfp-kubernetes-execution-tests`

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

connor-mccarthy

/lgtm
/approve

Thank you, Kevin!

google-oss-prow · 2024-02-06T23:03:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: connor-mccarthy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~sdk/OWNERS~~ [connor-mccarthy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…0441) * samples message * update pr number * Split LoopArgument into LoopParameterArgument and LoopArtifactArgument * formatting * address some comments * resolve release notes conflict * flatten loops in pipeline_spec_builder * update artifact type checking logic * simplify artifact checking logic * re-add issubtype_of_artifact * move name_is_loop_argument to for_loop_test.py * simplify LoopArtifactArgument is_artifact_list=False logic * update typeerror * typo * typo * small fix * formatting * formatting * remove issubtype_of_artifact() * small changes * assert LoopArtifactArgument channel.is_artifact_list is True * remove whitespace * remove newline * Update single artifact check and error message * formatting * add unit test for is_artifact_list==False * formatting * update valueerror test. * typo * regex formatting * formatting Signed-off-by: Alexey Roytman <roytman@il.ibm.com>

…0441) * samples message * update pr number * Split LoopArgument into LoopParameterArgument and LoopArtifactArgument * formatting * address some comments * resolve release notes conflict * flatten loops in pipeline_spec_builder * update artifact type checking logic * simplify artifact checking logic * re-add issubtype_of_artifact * move name_is_loop_argument to for_loop_test.py * simplify LoopArtifactArgument is_artifact_list=False logic * update typeerror * typo * typo * small fix * formatting * formatting * remove issubtype_of_artifact() * small changes * assert LoopArtifactArgument channel.is_artifact_list is True * remove whitespace * remove newline * Update single artifact check and error message * formatting * add unit test for is_artifact_list==False * formatting * update valueerror test. * typo * regex formatting * formatting

samples message

7145184

google-oss-prow bot requested review from chensun and connor-mccarthy January 29, 2024 21:55

google-oss-prow bot added the size/XL label Jan 29, 2024

KevinGrantLee mentioned this pull request Jan 29, 2024

feat(sdk): Support dsl.ParallelFor over list of Artifacts #10437

Closed

1 task

update pr number

8d4e7ed

KevinGrantLee marked this pull request as draft January 29, 2024 22:12

google-oss-prow bot added the do-not-merge/work-in-progress label Jan 29, 2024

Split LoopArgument into LoopParameterArgument and LoopArtifactArgument

241cf75

KevinGrantLee marked this pull request as ready for review January 30, 2024 00:41

google-oss-prow bot removed the do-not-merge/work-in-progress label Jan 30, 2024

formatting

c738cd4

connor-mccarthy requested changes Jan 30, 2024

View reviewed changes

KevinGrantLee added 8 commits January 30, 2024 15:29

address some comments

3fe31e8

resolve release notes conflict

3050e4c

Merge branch 'master' into parallelfor-list-artifacts

fa9027b

Merge branch 'master' of https://github.com/KevinGrantLee/pipelines i…

c8b65de

…nto parallelfor-list-artifacts

Merge branch 'parallelfor-list-artifacts' of https://github.com/Kevin…

3462a15

…GrantLee/pipelines into parallelfor-list-artifacts

flatten loops in pipeline_spec_builder

43ef672

update artifact type checking logic

e5d5c9b

simplify artifact checking logic

974be3a

connor-mccarthy requested changes Jan 31, 2024

View reviewed changes

KevinGrantLee added 8 commits January 31, 2024 14:28

re-add issubtype_of_artifact

f649fe8

move name_is_loop_argument to for_loop_test.py

4e2f033

simplify LoopArtifactArgument is_artifact_list=False logic

29bc8b1

update typeerror

9a31af8

typo

231aecb

typo

3090549

small fix

753115a

formatting

b4a01dd

google-oss-prow bot added the approved label Feb 5, 2024

Update single artifact check and error message

4337f87

KevinGrantLee requested a review from connor-mccarthy February 5, 2024 23:02

formatting

106f8b6

connor-mccarthy reviewed Feb 5, 2024

View reviewed changes

sdk/python/kfp/dsl/for_loop.py Show resolved Hide resolved

add unit test for is_artifact_list==False

d29d8cb

KevinGrantLee requested a review from connor-mccarthy February 5, 2024 23:43

formatting

d1141fa

connor-mccarthy requested changes Feb 6, 2024

View reviewed changes

sdk/python/kfp/dsl/for_loop_test.py Outdated Show resolved Hide resolved

google-oss-prow bot removed the approved label Feb 6, 2024

update valueerror test.

19e693c

KevinGrantLee requested a review from connor-mccarthy February 6, 2024 19:46

typo

564f666

connor-mccarthy reviewed Feb 6, 2024

View reviewed changes

sdk/python/kfp/dsl/for_loop_test.py Outdated Show resolved Hide resolved

KevinGrantLee added 2 commits February 6, 2024 13:49

regex formatting

11ea34f

formatting

d367372

KevinGrantLee requested a review from connor-mccarthy February 6, 2024 21:54

connor-mccarthy approved these changes Feb 6, 2024

View reviewed changes

google-oss-prow bot added the lgtm label Feb 6, 2024

google-oss-prow bot added the approved label Feb 6, 2024

google-oss-prow bot merged commit b528568 into kubeflow:master Feb 6, 2024
27 of 28 checks passed

KevinGrantLee deleted the parallelfor-list-artifacts branch February 6, 2024 23:04

KevinGrantLee mentioned this pull request Feb 28, 2024

test: Add ParallelFor compile test over single artifact. #10531

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(sdk): Support dsl.ParallelFor over list of Artifacts #10441

feat(sdk): Support dsl.ParallelFor over list of Artifacts #10441

KevinGrantLee commented Jan 29, 2024

google-oss-prow bot commented Feb 6, 2024

connor-mccarthy left a comment

google-oss-prow bot commented Feb 6, 2024

feat(sdk): Support dsl.ParallelFor over list of Artifacts #10441

feat(sdk): Support dsl.ParallelFor over list of Artifacts #10441

Conversation

KevinGrantLee commented Jan 29, 2024

google-oss-prow bot commented Feb 6, 2024

connor-mccarthy left a comment

Choose a reason for hiding this comment

google-oss-prow bot commented Feb 6, 2024