Support subdags for reusing pipelines

[IronPan]

This will allow reusing a subset of the pipeline in other pipelines, similar to the function
https://airflow.apache.org/docs/stable/concepts.html?highlight=subdag#subdags

[numerology]

This is very useful.

Also shall we consider how to make it sharable? For example we have component yaml spec today, should we have subdag yaml spec as well? @Ark-kun

[Ark-kun]

@IronPan This will allow reusing a subset of the pipeline in other pipelines

@numerology Also shall we consider how to make it sharable? For example we have component yaml spec today, should we have subdag yaml spec as well?

If I understood you correctly, the feature you ask for is already supported.
The feature is called "graph components".

Reuse has two aspects - creating and using.

The users can create a graph component from a pipeline-like function using create_graph_component_from_pipeline_func. This feature was added in v0.1.32: #2273

The users can also write the spec manually. See real e2e example. The specification and structures for the graph components have been present in KFP since the first G.A. release.

To use a graph component simply load it the same way as container components: using load_component_from_url or load_component_from_file. The user does not need to know whether the component is implemented as a single container or as a task graph.

Are there new clients that ask for this feature? (We can talk offline)
Are there any additional features that should be supported?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.