Client-side computation of cached steps #3068
Conversation
github-actions
bot
added
internal
schustmi
force-pushed
the
feature/PRD-657-precompute-cached-steps
branch
from
06341ec to
661869f
Compare
|
E2E template updates in |
|
Classification template updates in |
…zenml-io/zenml into feature/PRD-657-precompute-cached-steps
docs/book/user-guide/starter-guide/cache-previous-executions.md
Outdated
Show resolved
Hide resolved
Co-authored-by: Alexej Penner <thealexejpenner@gmail.com>
| data.pop("name", None) | ||
| data.pop("version", None) | ||
| data.pop("model", None) | ||
| return data |
There was a problem hiding this comment.
It looks like older pipelines might fail silently downstream because of this.
There was a problem hiding this comment.
In which case would they fail?
I'm guessing this would only be possible if upgrading the server to the new release while still having an old scheduled pipeline running that uses a Docker image of an older version? The server would then remove those attributes before sending the response.
Would that case be fixed if I just don't remove this data here and add extra=allow in the pydantic config?
There was a problem hiding this comment.
To me, it would be better just to give some instructions on how to fix this and write that deprecation clearly in the new release notes. I don't ask for any technical solution here - these guys should have been gone for a while already.
There was a problem hiding this comment.
The problem here is that these attributes will already be removed on the server side, which means the warning/instructions will be very hidden. I instead added some instructions on the runtime error below that happens if the ID is missing.
Co-authored-by: Andrei Vishniakov <31008759+avishniakov@users.noreply.github.com>
Describe changes
This PR adds functionality to compute cached steps client-side. This reduces the time/money spent waiting for remote orchestrators, which can often take a long time to spin up a step container only for it to immediately shut down because the step was cached.
Some things were removed as part of this PR to make the model linking more consistent for cached and non-cached steps:
ArtifactConfigPre-requisites
Please ensure you have done the following:
developand the open PR is targetingdevelop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.Types of changes