feat(sdk): add PipelineConfig to DSL to re-implement pipeline-level config

[gregsheremeta]

Description of your changes:

KFP v1 supported setting pipeline-level configuration via a PipelineConf class. This class was deprecated and no replacement was added to KFP v2.

Add new PipelineConfig class to support setting pipeline-level configuration in KFP v2.

Note: I discussed this approach with @chensun in the KFP Community meeting on August 14 (agenda, recording). He gave preliminary approval for this approach.

Here's an example of how it would be used.

Followup to #11333.

Checklist:

• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[DharmitD]

/rerun-all

[gregsheremeta]

@chensun bump to the top of your inbox :) thanks!

[gregsheremeta]

@chensun bump, looking for an ack on the approach and the proto fields :)

[gregsheremeta]

@chensun bumping, can you take a look?

[gregsheremeta]

Sharing the document that @chensun provided in today's KFP Community meeting (agenda, recording) :

[External] Notes - Discuss PipelineConfig - Google Docs
https://docs.google.com/document/d/1Av9sWpluH9KhP2hAgpNRCJhqN_xI3Cb8bfaIeNf6ltk/edit#heading=h.lngiup14o4uj

[gregsheremeta]

Thanks for discussing this internally. Here are my responses.

Actions items:
Ask community contributors to provide real world use cases and override designs for further evaluation.

Some examples:

• Allowing users to do synchronization at the pipeline level (issue, Argo Workflows doc)
• Setting TTL at the pipeline level (issue, example from WIP: feat(api/sdk/backend): implement PipelineConfig and setting ttl on pipelines. Fixes: #10899 #10942)
• Allowing users to control caching behavior at the pipeline level instead of on every task (issue)

Ask community contributors to evaluate PlatformSpec as an alternative place to add PipelineConf

I'm fine to put it wherever you all think is best. To clarify, is PlatformSpec the second document at the bottom?

root:
  dag:
    tasks:
      comp:
        cachingOptions:
          enableCache: true
        componentRef:
          name: comp-comp
        taskInfo:
          name: comp
schemaVersion: 2.1.0
sdkVersion: kfp-2.7.0
---
platforms:
  kubernetes:
    deploymentSpec:
      executors:
        exec-comp:
          emptyDirMounts:
          - medium: Memory
            mountPath: /mnt/my_vol_1
            sizeLimit: 1Gi
            volumeName: emptydir-vol-1

And if so, about where should pipeline-level stuff go? Should it be a sibling of kubernetes? Note that this was my approach in #10942 -- quoting from that

# PIPELINE DEFINITION
# Name: test-ttl-pipeline
components:
  comp-empty-component:
    executorLabel: exec-empty-component
deploymentSpec:
  completedPipelineTtl: 120
SNIP

but I (perhaps mistakenly) thought that that was rejected.

[gregsheremeta]

@chensun bumping to the top of your inbox. Can you provide guidance on the spec questions in my previous comment?

[rimolive]

@chensun As we discussed in the last Pipelines WG meeting, we need your inputs about Greg comments.

[chensun]

Apologize for the slow response.

I'm fine to put it wherever you all think is best. To clarify, is PlatformSpec the second document at the bottom?

Correct

And if so, about where should pipeline-level stuff go? Should it be a sibling of kubernetes? Note that this was my approach in #10942 -- quoting from that

kubernetes is meant for all features generic to kubernetes, the other platform exists today is google_cloud. So if we think a feature can be applied to all kubernetes-based platform (including Argo, Tekton), we can add the feature under it. Otherwise, if there's an Argo-specific feature that does not apply to KFP-Tekton, we may consider creating a sibling to kubernetes as argo. For TTL, I think it should be under kubernetes.

In terms of where specifically, pipeline level config doesn't exists yet. I think we may want to add it as a sibling to the deployment_config, which is the component-level:

pipelines/api/v2alpha1/pipeline_spec.proto

Line 1084 in 8590ec1

PlatformDeploymentConfig deployment_spec = 1;

This would needs some proper design.

[gregsheremeta]

@chensun confirming what we talked about on the call today. Hypothetically, say we're adding a feature to do both pipeline-level and pod-level TTL configs. We agreed those are not argo-specific or tekton-specific -- they belong in "kubernetes". So they would look like this:

<SNIP>
schemaVersion: 2.1.0
sdkVersion: kfp-2.8.0
---
platforms:
  kubernetes:
    pipelineConfig:
      pipelineTtl: "120"
    deploymentSpec:
      executors:
        exec-my-component:
          podTtl: "240"

Look right to you?

[chensun]

@chensun confirming what we talked about on the call today. Hypothetically, say we're adding a feature to do both pipeline-level and pod-level TTL configs. We agreed those are not argo-specific or tekton-specific -- they belong in "kubernetes". So they would look like this:

<SNIP>
schemaVersion: 2.1.0
sdkVersion: kfp-2.8.0
---
platforms:
  kubernetes:
    pipelineConfig:
      pipelineTtl: "120"
    deploymentSpec:
      executors:
        exec-my-component:
          podTtl: "240"

Look right to you?

Yes!

[rimolive]

/lgtm

[DharmitD]

Tested these changes locally and verified these work as expected.

/lgtm

[HumairAK]

Thanks folks.

/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: HumairAK

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• sdk/OWNERS [HumairAK]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sridhar1029]

@gregsheremeta I see that this change was merged but nothing was added to the Pipeline level config (sdk/python/kfp/dsl/pipeline_config.py)
Is there a plan to add config options in later versions? something like example you talked about (#11113)

[jgarciao]

@sridhar1029 I think the ttl configuration flag will be added in this follow-up task #10899