Need feature parity between DSL and component.yaml.

[alicelee0606]

When trying to build a reusable component, there are some attributes that are originally supported by ContainerOp or Kubernetes container v1 core but are not supported in implementation.container in component.yaml (ex: output_artifact_paths/volumes/image_pull_policy...). Wondering what is the best practice for documenting these for each component? Or building a class component is more suggested than using component.yaml in this case?

[alicelee0606]

Hi @vicaire
Thanks for labeling this issue. But I guess it is more of an area/sdk or area/sdk/dsl issue?

[Ark-kun]

there are some attributes that are originally supported by ContainerOp or Kubernetes container v1 core but are not supported in implementation.container in component.yaml (ex: output_artifact_paths/volumes/image_pull_policy...).

The intent of components to be portable and reusable. Some features that would make the component non-portable (e.g. volumes) are deliberately absent.
The idea is to customize those features on the task level (task = component + arguments + customization). So it's by design that when you want attach some particular volume (which is non-portable object since it's cluster specific), you need to do that on the task level or pipeline level.

specific_task1 = portable_op(...).add_volume_mount('volume 1')
specific_task2 = portable_op(...).add_volume_mount('volume 2')

There is also the issue of separation of concerns between the component author and the pipeline author. Should component dictate its own image pull policy or RAM usage? That's probably in the pipeline author's jurisdiction.

So, when building a reusable component think about what would be common between all invocations. This common part should be captured in the component definition (component.yaml). All other things that can be different should be provided when the pipeline author uses the comonent either in form of arguments or in form of tweaks like .apply, .set_resource_limit etc.

output_artifact_paths

The support for user artifacts is not yet complete.
There is some preliminary support for declaring output artifacts (output_artifact_paths), but not declaring input artifacts and not artifact passing. I'm currently working on introducing this support.

My current vision is that there should be no distinction between output parameters and output artifacts. All output data blobs should be stored in a permanent storage (like artifacts) and then some of them will be consumed as files/artifacts and some will be consumed as parameters.

When this move occurs, you'll be able to specify the output artifacts the same way you specify the output parameters now - {outputPath: My output} (or the legacy fileOutputs that should be avoided).

[alicelee0606]

Yes. I think that makes sense. Thanks for the response and looking forward to more complete output data supports!

[Ark-kun]

@alicelee0606
BTW, I'd be glad to help you with building reusable components. I'm often present on the Kebeflow slack channel. Our documentation might be a bit convoluted sometimes. https://www.kubeflow.org/docs/pipelines/sdk/component-development/