Passing pipeline parameters as yaml config #2058

avishniakov · 2023-11-17T09:40:38Z

Describe changes

Add ability to pass in pipeline params as yaml config:

parameters:
  environment: production

Also added a RuntimeError for the cases when runtime config conflicts with YAML config (both steps and pipeline params) - @schustmi please confirm that this is acceptable or shall I revert to config-dominant silent mode.
Docs: https://app.gitbook.com/o/-MCl1Hlw9oU4xibZ-ymf/s/9oLgDi1mkUbwQiTmrEit/user-guide/advanced-guide/pipelining-features/configure-steps-pipelines#parameters-for-your-steps

BREAKING CHANGES

What passed silently before now throws an error (step/pipeline params configured in file and code differently)

# config.yaml
parameters:
    some_param: 24

steps:
  my_step:
    parameters:
      input_2: 42

# run.py
@step
def my_step(input_1: int, input_2: int) -> None:
    pass

@pipeline
def my_pipeline(some_param: int):
    # here an error will be raised since `input_2` is
    # `42` in config, but `43` was provided in the code
    my_step(input_1=42, input_2=43)

if __name__=="__main__":
    # here an error will be raised since `some_param` is
    # `24` in config, but `23` was provided in the code
    my_pipeline(23)

Pre-requisites

Please ensure you have done the following:

I have read the CONTRIBUTING.md document.
If my change requires a change to docs, I have updated the documentation accordingly.
If I have added an integration, I have updated the integrations table and the corresponding website section.
I have added tests to cover my changes.
I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Other (add details above)

avishniakov · 2023-11-17T10:24:54Z

@schustmi , these changes clash a bit with what you created in #1638.
Can I have your perspective on this? I made an error to be raised if a user provides confusing configurations (step input is set in code and config), but before config value just dominated it. WDYT, shall I revert to the old behavior or update the test case instead?

…ers-as-yaml-and-document

github-actions · 2023-11-22T11:42:12Z

E2E template updates in examples/e2e have been pushed.

…ers-as-yaml-and-document

github-actions · 2023-11-23T18:02:35Z

E2E template updates in examples/e2e have been pushed.

…ers-as-yaml-and-document

avishniakov · 2023-12-05T12:35:16Z

@bcdurak @schustmi can you have a look?

schustmi · 2023-12-06T09:22:01Z

src/zenml/new/pipelines/pipeline.py

+    step_name()
+
+if __name__=="__main__":
+    pipeline_(param_name="value2")


This would also need to call with_options(config_file=...) to showcase the actual error right?

schustmi · 2023-12-06T09:23:27Z

src/zenml/new/pipelines/pipeline.py

+            parameters_ = dict_utils.recursive_update(parameters_, from_file_)
+        if parameters_:
+            for k, v_runtime in kwargs.items():
+                if v_config := parameters_.get(k, None):


None or False might also be valid parameter values which will not get detected by this logic I think?

schustmi · 2023-12-06T09:24:33Z

src/zenml/new/pipelines/pipeline.py

+        if parameters_:
+            for k, v_runtime in kwargs.items():
+                if v_config := parameters_.get(k, None):
+                    conflicting_parameters[k] = (v_config, v_runtime)


In the step you added an additional check if the config and runtime parameter are different, I think that also makes sense here?

schustmi · 2023-12-06T09:25:22Z

src/zenml/steps/base_step.py

        for key, value in parameters.items():
+            if runtime_value := runtime_parameters.get(key, None):


Same as for the pipeline, None/False/0/... as parameter values will probably not be included in these checks.

…ers-as-yaml-and-document

bcdurak

Great changes and it is nice that you added the explanation to the docs as well. From what I can see all seems good 👍

enable passing pipeline params as YAML

6293327

avishniakov changed the base branch from main to develop November 17, 2023 09:40

github-actions bot added internal To filter out internal PRs and issues enhancement New feature or request labels Nov 17, 2023

avishniakov added 4 commits November 17, 2023 11:12

add docs

1b343b4

add docs

134c6f3

add docs

91f6794

add docs

e1f39a2

raise on pipeline param conflict

d7a96fe

avishniakov requested review from bcdurak and schustmi November 17, 2023 10:38

avishniakov changed the title ~~Feature/oss 2529 passing pipeline parameters as yaml and document~~ Passing pipeline parameters as yaml config Nov 17, 2023

avishniakov added the breaking-change label Nov 17, 2023

avishniakov and others added 4 commits November 17, 2023 14:14

improve docs

6724f17

Merge branch 'develop' into feature/OSS-2529-passing-pipeline-paramet…

2469bd7

…ers-as-yaml-and-document

adjust test to new behavior

59eaf13

Auto-update of E2E template

53099ca

avishniakov and others added 3 commits November 23, 2023 09:38

Merge branch 'develop' into feature/OSS-2529-passing-pipeline-paramet…

a5a7ba4

…ers-as-yaml-and-document

Merge branch 'develop' into feature/OSS-2529-passing-pipeline-paramet…

715b03d

…ers-as-yaml-and-document

Auto-update of E2E template

b040823

Merge branch 'develop' into feature/OSS-2529-passing-pipeline-paramet…

063b4ba

…ers-as-yaml-and-document

schustmi requested changes Dec 6, 2023

View reviewed changes

avishniakov added 2 commits December 6, 2023 13:08

comments from review

8a7a10d

Merge branch 'develop' into feature/OSS-2529-passing-pipeline-paramet…

8f8e64a

…ers-as-yaml-and-document

bcdurak approved these changes Dec 8, 2023

View reviewed changes

avishniakov requested a review from schustmi December 8, 2023 14:02

avishniakov merged commit d5a3aef into develop Dec 11, 2023
30 checks passed

avishniakov deleted the feature/OSS-2529-passing-pipeline-parameters-as-yaml-and-document branch December 11, 2023 15:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Passing pipeline parameters as yaml config #2058

Passing pipeline parameters as yaml config #2058

avishniakov commented Nov 17, 2023 •

edited

Loading

avishniakov commented Nov 17, 2023 •

edited

Loading

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 23, 2023

avishniakov commented Dec 5, 2023

schustmi Dec 6, 2023

avishniakov Dec 6, 2023

schustmi Dec 6, 2023

avishniakov Dec 6, 2023

schustmi Dec 6, 2023

avishniakov Dec 6, 2023

schustmi Dec 6, 2023

avishniakov Dec 6, 2023

bcdurak left a comment

		for key, value in parameters.items():
		if runtime_value := runtime_parameters.get(key, None):

Passing pipeline parameters as yaml config #2058

Passing pipeline parameters as yaml config #2058

Conversation

avishniakov commented Nov 17, 2023 • edited Loading

Describe changes

Pre-requisites

Types of changes

avishniakov commented Nov 17, 2023 • edited Loading

github-actions bot commented Nov 22, 2023

github-actions bot commented Nov 23, 2023

avishniakov commented Dec 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bcdurak left a comment

Choose a reason for hiding this comment

avishniakov commented Nov 17, 2023 •

edited

Loading

avishniakov commented Nov 17, 2023 •

edited

Loading