Wait for model directory to exist when "lazy_load_local_models" is enabled

[rhuss]

Is your feature request related to a problem? Please describe.

With the new modelcar feature of KServe it is possible to access model data directly from within an OCI image without downloading or copying. However, since this model is injected as a sidecar, the startup order of the containers in non-deterministic.

Since the modelcar container creates the symbolic link /mnt/models to point to the model stored within that image, the path /mnt/models might not exist when the caikit transformer container starts. This will trigger line

caikit/caikit/runtime/model_management/model_manager.py

Line 105 in 4b42d37

if self._local_models_dir and not os.path.exists(self._local_models_dir):

so that the startup fails.

Describe the solution you'd like

When lazy loading is enabled via lazy_load_local_models, the runtime should wait a certain amount of time for the path (/mnt/models) to come up before giving up. This can be tricky since when the model still needs to be loaded in the node's OCI runtime, it gets pulled from the registry, which might take quite some time. But if it is already loaded, the model can be delayed in a matter of seconds.

Additional context

More about the modelcar approach can be found here:

• Feature Track: https://docs.google.com/document/d/1Bs4fnP8rhPMaoPoLSYVvuRq-z9vkGPQ0rKbmfH4I7js/edit?usp=sharing
• PR: Add support for modelcars kserve/kserve#3110
• Short Demo: https://youtu.be/KzWH8v6CcR0?si=tZTaMbkrAC-eWqQg
• Original Issue: Enhancing KServe Model Fetching with oci: Schema and Sidecar Containers kserve/kserve#3043