Support pipeline parameter definition by interacting with dsl.Pipeline

[numerology]

Currently the only way to define pipeline params in KFP dsl is through the signature of the pipeline function. A more systematic and dynamic way is needed to support use cases where the set of pipeline parameters are passed from external.

Example:
TFX's KFP dag runner

[merlintang]

do you mean that we can run a pipeline by passing the parameter into pipeline? for example, we load a pipeline file, then run the pipeline with new parameters? I have done this before, I can send a pr for this.

[numerology]

do you mean that we can run a pipeline by passing the parameter into pipeline? for example, we load a pipeline file, then run the pipeline with new parameters? I have done this before, I can send a pr for this.

Actually a bit of different. Currently through the signature of the pipeline function, we can define pipeline parameters. However, pipeline function signature is immutable in itself. If we further need to specify the pipeline parameter (not only their runtime values) in a more programmatical way, some alternatives are needed.

[merlintang]

I see, you mean that user can add more parameters to the pipeline ?

[numerology]

Yes. But this feature is not of high priority, though.

…

On Mon, Sep 9, 2019, 10:56 PM Mingjie Tang ***@***.***> wrote: I see, you mean that user can add more parameters to the pipeline ? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1942?email_source=notifications&email_token=ACJIYGRWMJJYKWFDVUAJSE3QI4ZIJA5CNFSM4IPEMPHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6J5NUY#issuecomment-529782483>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACJIYGTVBVJKWANOWB2KYB3QI4ZIJANCNFSM4IPEMPHA> .

[Ark-kun]

dsl.Pipeline is a context class that's supposed to be compiler-internal. We were planning to take it private.
Did you mean the dsl.pipeline decorator?

[numerology]

dsl.Pipeline is a context class that's supposed to be compiler-internal. We were planning to take it private.
Did you mean the dsl.pipeline decorator?

That would be ideal if we can support that without interacting with dsl.Pipeline. However I couldn't think of an easy way to that.
If we can introduce a field to specify pipeline parameter in the decorator, still we need to think carefully how we are going to use that in each component.

[numerology]

dsl.Pipeline is a context class that's supposed to be compiler-internal. We were planning to take it private.
Did you mean the dsl.pipeline decorator?

Let me give another look. I think we have some publc APIs that can be used currently. Maybe I don't need to directly manipulate dsl.Pipeline

[gaoning777]

Are we planning to take dsl.Pipeline private? If so, why?
Exposing a Pipeline class similar to the TFX Pipeline is not a bad idea to me. Strongly typed, support dynamic parameters, dynamic pipeline configurations(

pipelines/sdk/python/kfp/dsl/_pipeline.py

Line 54 in a4fa1ed

class PipelineConf():

).
Of course, having a simple interface and having the SDK generate the dsl.Pipeline is great for new users. However, exposing the full control of the Pipeline can unblock advanced users for custom scenarios.

[numerology]

Are we planning to take dsl.Pipeline private? If so, why?
Exposing a Pipeline class similar to the TFX Pipeline is not a bad idea to me. Strongly typed, support dynamic parameters, dynamic pipeline configurations(

pipelines/sdk/python/kfp/dsl/_pipeline.py

Line 54 in a4fa1ed

class PipelineConf():

).
Of course, having a simple interface and having the SDK generate the dsl.Pipeline is great for new users. However, exposing the full control of the Pipeline can unblock advanced users for custom scenarios.

I'm good with either private or public dsl.Pipeline. However I feel like we need a way to do something like (tentatively, did not carefully think it through)
author_pipeline(
pipeline_func, # where ContainerOp is initialized, perhaps with PipelineParam in its args.
pipeline_name,
pipeline_description,
pipeline_params)

[merlintang]

thanks @num

[numerology]

Added an API in #2082, which allows users to generate the workflow spec of the pipeline in a programmable way.

Marked as fixed for now.