Incorporate runtime into model configuration #285
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kthui
force-pushed
the
jacky-python-based-pytorch
branch
from
ae1bc12
to
368aa6b
Compare
kthui
force-pushed
the
jacky-python-based-pytorch
branch
2 times, most recently
from
50932e1
to
5dd3ac5
Compare
kthui
force-pushed
the
jacky-python-based-pytorch
branch
from
5dd3ac5
to
3959a19
Compare
nnshah1
reviewed
Nov 14, 2023
rmccorm4
reviewed
Nov 27, 2023
rmccorm4
reviewed
Nov 27, 2023
rmccorm4
reviewed
Nov 27, 2023
rmccorm4
reviewed
Nov 28, 2023
nnshah1
reviewed
Nov 28, 2023
nnshah1
reviewed
Nov 28, 2023
nnshah1
reviewed
Nov 28, 2023
src/constants.h
Outdated
| constexpr char kPyTorchBackend[] = "pytorch"; | ||
|
|
||
| constexpr char kPythonFilename[] = "model.py"; | ||
| constexpr char kPythonBackend[] = "python"; | ||
|
|
||
| constexpr char kVLLMBackend[] = "vllm"; |
There was a problem hiding this comment.
Do we need these for any new python based backend? Would be good to avoid that if possible - we shouldn't need to update the constants if someone adds a backend in the future
There was a problem hiding this comment.
We do not need them. constexpr char kVLLMBackend[] = "vllm"; is removed, and they are handled as custom backend.
kthui
force-pushed
the
jacky-python-based-pytorch
branch
2 times, most recently
from
df1f9ff
to
0d89719
Compare
kthui
force-pushed
the
jacky-python-based-pytorch
branch
from
0d89719
to
b0e4663
Compare
…jacky-python-based-pytorch
…jacky-python-based-pytorch
…jacky-python-based-pytorch
rmccorm4
reviewed
Jan 10, 2024
rmccorm4
previously approved these changes
Jan 10, 2024
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
rmccorm4
approved these changes
Jan 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related PRs:
Previously, the runtime for each backend is determined automatically when loading the backend. The PR adds the ability for user to choose which runtime to use for each model on the model configuration. For example:
To retain backward compatibility, the logic previously used to determine runtime is incorporated into autocomplete, which will attempt to fill-in the "runtime" field if left empty. The core will load the exact runtime as specified on the "runtime" field on the model configuration, after any applicable autocomplete.
As a part of the change, the autocomplete logic will fill in Python backend based runtime for backends that are clearly Python backend based backends, for example, vLLM backend.
If a backend provides both C++ and Python backend based runtime, for example, PyTorch backend, the autocomplete will look at the default model filename on the model configuration to try determine whether the C++ or Python backend based runtime is more appropriate. For example, if the default model filename on PyTorch backend is "model.pt", the C++ runtime will be selected. If the default model filename is "model.py", the Python backend based runtime will be selected. In any case, the autocomplete will not alter the runtime if it is explicitly provided by the user.