Add script to wait for the model to appear when using KServe Modelcar #141
Conversation
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: israel-hdez The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
israel-hdez
force-pushed
the
modelcar-waiting
branch
from
12795f9
to
c36acc7
Compare
When using OCI containers for model storage in KServe (modelcar), there is the possibility that the model server starts before the model has been fully downloaded. When this happens, the model server would terminate with error because the model path is empty. This adds a small script to wait for the cluster to fully download the model container before invoking the model server. The waiting is triggered when the MODEL_INIT_MODE environment variable is set to the "async". Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com>
israel-hdez
force-pushed
the
modelcar-waiting
branch
from
c36acc7
to
4931804
Compare
|
@israel-hdez: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
|
||
| if [ "${MODEL_INIT_MODE}" = "async" ] ; then | ||
| echo "Waiting for model files (modelcar) to be present..." | ||
| until test -e /mnt/models; do |
There was a problem hiding this comment.
Should we have a (configurable) timeout here?
|
We discussed in a call, and this won't be needed. We are changing kserve-controller to reduce the race condition. |
* Dockerfile.rocm bug fix * naming preference --------- Co-authored-by: Gregory Shtrasberg <156009573+gshtras@users.noreply.github.com>
When using OCI containers for model storage in KServe (modelcar), there is the possibility that the model server starts before the model has been fully downloaded. When this happens, the model server would terminate with error because the model path is empty.
This adds a small script to wait for the cluster to fully download the model container before invoking the model server. The waiting is triggered when the MODEL_INIT_MODE environment variable is set to the "async".