[FeatureRequest]KFP execution cache

[rui5i]

1. Mutate webhook

• webhook service
• execution_key generator
• pod modification controller
• max_cache_staleness support

2. Mysql

• persist execution_key

3. Releasing

• standalone deployment
• mkp deployment

[rui5i]

/assign @rui5i

[rmgogogo]

not sure which is the first phase target, from SDK-to-API, right?

[rui5i]

Sorry for the confusing. Update the issue content. We change the project to "KFP execution cache" and retry from a certain step will become a underlying CUJ. There will not be any new APIs. The design doc will be sent out really soon.

[elikatsis]

Hello @rui5i,

That's a really nice feature!
Any update on the design doc? I think it should be an item in the upcoming KFP community meeting.

Thanks!

[rui5i]

Hi @elikatsis ,

Thanks for your feedback! The design doc currently is under KFP team review. I am happy to present on the upcoming KFP community meeting!

[elikatsis]

Ping!
I see there is no entry in the agenda. Any update on that?

[Padarn]

Also interested in any docs on this, or updates on the release. I saw some meeting notes here: https://docs.google.com/document/d/1KB5KD8TvcrnxQX0xluHRnUdRYkM2-5-vB2V1wlxS4GY/edit#heading=h.gt0qfhljl8xo but its not clear to me if a decision has been made

[rui5i]

Also interested in any docs on this, or updates on the release. I saw some meeting notes here: https://docs.google.com/document/d/1KB5KD8TvcrnxQX0xluHRnUdRYkM2-5-vB2V1wlxS4GY/edit#heading=h.gt0qfhljl8xo but its not clear to me if a decision has been made

Hi, thanks for asking! The link you provided is our caching design doc. We are trying to make it available on 0.3.1. I'll let you know after it's release.

[Padarn]

Awesome. Feel free to ask if there is anything can help from community.

[elikatsis]

Hi again,
nice job with the implementation so far! Keep it up!

I wanted to ask, are there any plans on what will you be showing as step's logs in KFP UI? Or the cached steps will appear with empty logs?
I haven't spotted any change related to that, so I assume, correct me if I'm wrong, that if we were to deploy the feature and cache a step at this very moment, logs would be empty.

[rui5i]

Hi again,
nice job with the implementation so far! Keep it up!

I wanted to ask, are there any plans on what will you be showing as step's logs in KFP UI? Or the cached steps will appear with empty logs?
I haven't spotted any change related to that, so I assume, correct me if I'm wrong, that if we were to deploy the feature and cache a step at this very moment, logs would be empty.

Hi @elikatsis ,

Thanks for checking in! Currently, if a step's result is taken from cache, then the step log will show "This step output is taken from cache." https://github.com/kubeflow/pipelines/blob/master/backend/src/cache/server/mutation.go#L119. In the future we may explore if it's possible to show the link of previous run/step.

[rui5i]

Kubeflow Pipelines step caching is now released in 0.4.0 and after. Close this issue.

[Bobgy]

/reopen

TODO:

• Add an UI indication a step is cached [UI] Show cached steps #3602
  /assign @Bobgy

I will finish the UI integration

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen

TODO:

• Add an UI indication a step is cached
  /assign @Bobgy

I will finish the UI integration

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Bobgy]

@rui5i @Ark-kun I took a look at a cached pod and read through the mutation hook's code, it seems we don't have any information UI can use to indicate a pod is cached. Is that correct?

If so, where should we add these information?

[Ark-kun]

Looks like we did not add any extra labels, so the labels/annotations of reused pods are same.
We could add a few annotations here: https://github.com/kubeflow/pipelines/blob/master/backend/src/cache/server/mutation.go#L126 For example we can include the original start time and end time.

However, given a pod, it's pretty easy to detect that it was skipped (it does not have any Argo containers):

pipelines/backend/src/cache/server/mutation.go

Line 133 in c4fb794

Command: []string{`echo`, `"This step output is taken from cache."`},

There is also a way to detect the skipped pods based on the WorkflowStatus alone: All output artifacts have the pod name in the URI. But for skipped pods, the pod name does not match the URIs. This genius idea belongs to @rui5i. (And now there are always some output artifacts since we've enabled log archiving).

[Bobgy]

@Ark-kun I'm a little worried about if we are relying too much on argo details:

• we need to let UI understand argo artifact path structure, there is a part that's the workflow name
• UI needs to understand cache server modifies argo outputs to reuse past outputs
• There are no other systems modifying argo outputs

The last two points probably is also dangerous to cache server...

It really sounds to me we should contribute the caching solution to argo workflow natively and add it to workflow status. (just personal gut feeling, I guess that's not practical now)

In the mean time, I'll let UI use the hack first to first get it working.

[Bobgy]

Digging through some related argo issues, I understand argo isn't really taking the feature request.

[Bobgy]

@Ark-kun Can I assume we only cache successful steps?

[Ark-kun]

It really sounds to me we should contribute the caching solution to argo workflow natively and add it to workflow status. (just personal gut feeling, I guess that's not practical now)

I'm positive about that. Half a year ago I even started the project to add caching to Argo, but did not finish it. The caching requires some persistence and Argo did not have any DB at that time.

@Ark-kun Can I assume we only cache successful steps?

We only reuse the successfull steps.

In the future we'll start reusing still-running steps.