Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Runtime reference post-release edits #430

Merged
merged 6 commits into from
Aug 28, 2024
Merged

Runtime reference post-release edits #430

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
704dbc7
Post-release edits
eturner24 Aug 27, 2024
3a5fbe9
Formatting
eturner24 Aug 27, 2024
52da7ca
Combining notes
eturner24 Aug 27, 2024
4d2befc
Correct links
eturner24 Aug 28, 2024
e88d126
Peer review
eturner24 Aug 28, 2024
fdda4d7
Peer review part 2
eturner24 Aug 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 8 additions & 30 deletions ...ing-inference-endpoint-for-model-deployed-on-single-model-serving-platform.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,13 @@
[role='_abstract']
To make inference requests to your deployed model, you must know how to access the inference endpoint that is available.

ifdef::upstream[]
For a list of paths to use with the supported runtimes and example commands, see link:{odhdocshome}/serving-models/#inference-endpoints_serving-large-models[Inference endpoints].
endif::[]
ifndef::upstream[]
For a list of paths to use with the supported runtimes and example commands, see link:{rhoaidocshome}{default-format-url}/serving_models/serving-large-models_serving-large-models#inference-endpoints_serving-large-models[Inference endpoints].
endif::[]
eturner24 marked this conversation as resolved.
Show resolved Hide resolved

.Prerequisites
* You have logged in to {productname-long}.
ifndef::upstream[]
Expand All @@ -21,38 +28,9 @@ endif::[]
. From the {productname-short} dashboard, click *Model Serving*.
+
The inference endpoint for the model is shown in the *Inference endpoint* field.
. Depending on what action you want to perform with the model (and if the model supports that action), copy the inference endpoint shown and then add a path to the end of the URL.

ifdef::upstream[]
[NOTE]
====
For a list of paths to use with the supported runtimes, see link:{odhdocshome}/serving-models/#inference-endpoints_serving-large-models[Inference endpoints].
====
endif::[]

ifndef::upstream[]
[NOTE]
====
For a list of paths to use with the supported runtimes, see link:{rhoaidocshome}{default-format-url}/serving_models/serving-large-models_serving-large-models#inference-endpoints[Inference endpoints].
====
endif::[]

. Depending on what action you want to perform with the model (and if the model supports that action), copy the inference endpoint and then add a path to the end of the URL.
. Use the endpoint to make API requests to your deployed model.

ifdef::upstream[]
[NOTE]
====
For a list of sample commands, see link:{odhdocshome}/serving-models/#inference-endpoints_serving-large-models[Inference endpoints].
====
endif::[]

ifndef::upstream[]
[NOTE]
====
For a list of sample commands, see link:{rhoaidocshome}{default-format-url}/serving_models/serving-large-models_serving-large-models#inference-endpoints[Inference endpoints].
====
endif::[]

[role='_additional-resources']
.Additional resources
* link:https://github.com/IBM/text-generation-inference[Text Generation Inference Server (TGIS)^]
Expand Down
6 changes: 3 additions & 3 deletions modules/ref-inference-endpoints.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,8 +212,8 @@ endif::[]
* link:https://docs.openvino.ai/2023.3/ovms_docs_rest_api_kfs.html[OpenVINO KServe-compatible REST API documentation^]
* link:https://platform.openai.com/docs/api-reference/introduction[OpenAI API documentation^]
ifdef::upstream[]
* link:{odhdocshome}/serving-models/#ref-supported-runtimes_serving-large-models[Supported runtimes]
* link:{odhdocshome}/serving-models/#supported-runtimes_serving-large-models[Supported model-serving runtimes]
endif::[]
ifndef::upstream[]
* link:{rhoaidocshome}{default-format-url}/serving_models/serving-large-models_serving-large-models#ref-supported-runtimes[Supported runtimes]
endif::[]
* link:{rhoaidocshome}{default-format-url}/serving_models/serving-large-models_serving-large-models#supported-model-serving-runtimes_serving-large-models[Supported model-serving runtimes].
endif::[]
4 changes: 2 additions & 2 deletions modules/ref-supported-runtimes.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ endif::[]

| OpenVINO Model Server | REST | None | Yes | Yes | Raw and serverless

| Text Generation Inference Server (TGIS) Standalone ServingRuntime for KServe (3) | gRPC | None | No | Yes | Raw and serverless
| Text Generation Inference Server (TGIS) Standalone ServingRuntime for KServe | gRPC | None | No | Yes | Raw and serverless

| vLLM ServingRuntime for KServe | REST | None | No | Yes | Raw and serverless

Expand All @@ -73,6 +73,6 @@ ifdef::upstream[]
endif::[]

ifndef::upstream[]
* link:{rhoaidocshome}{default-format-url}/serving_models/serving-large-models_serving-large-models#inference-endpoints[Inference endpoints]
* link:{rhoaidocshome}{default-format-url}/serving_models/serving-large-models_serving-large-models#inference-endpoints_serving-large-models[Inference endpoints]
endif::[]