Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add embeddings output to model builder #1127

Merged
merged 10 commits into from
Dec 11, 2024
Merged

Conversation

kunal-vaishnavi
Copy link
Contributor

Description

This PR adds support for outputting the last hidden state in addition to the logits in ONNX models. Users can run their models with ONNX Runtime GenAI and use the generator's GetOutput API to obtain the hidden states.

C/C++:

std::unique_ptr<OgaTensor> embeddings = generator->GetOutput("hidden_states");

C#:

using var embeddings = generator.GetOutput("hidden_states");

Java:

Tensor embeddings = generator.getOutput("hidden_states");

Python:

embeddings = generator.get_output("hidden_states")

Motivation and Context

In SLMs and LLMs, the last hidden state represents a model's embeddings for a particular input before the language modeling head is applied. Generating embeddings for a model is a popular task. These embeddings can be used for many scenarios such as text classification, sequence labeling, information retrieval using retrieval-augmented generation (RAG), and more.

This PR helps the following issues:

src/config.h Outdated Show resolved Hide resolved
@kunal-vaishnavi kunal-vaishnavi merged commit c61aaa6 into main Dec 11, 2024
12 of 13 checks passed
@kunal-vaishnavi kunal-vaishnavi deleted the kvaishnavi/hidden-states branch December 11, 2024 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants