Support Alibaba-NLP/gte-Qwen2-7B-instruct embedding Model

[zhaochenyang20]

Motivation

Current SGLang only supports the e5-mistral embedding model. I added Alibaba-NLP/gte-Qwen2-7B-instruct model in this PR.

Also, previously SGLang determines a model as an embedding model through its hf_config.architectures. But gte model has the same architecture as CausalLM. So I added a new parameter in the server_args and changed the forward function of Qwen2ForCausalLM.

Modifications

1. Changed the forward function of Qwen2ForCausalLM.
2. Added a new parameter is_embedding in server_args.
3. Some related changes.
4. Added unit tests for gte models. (both in the generation and embedding tests. I used rouge-L score in the generation tests)
5. Changed readme.

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[zhaochenyang20]

@Ying1123 I added gte in the generation model test. Note that I changed the prefill tolerance accordingly and added the rouge-l metric instead of assert output_strs exactly the same.

[hnyls2002]

Why would this be needed?

[llmforever]

@zhaochenyang20
您好，使用您提交的这个方法，和原始transformer与sentence transformer得到的embedding差距都很大，并且效果不好，您能帮忙看看吗？7B/1.5B的模型都测试过了
different result compare with orginal transformer backend,why?

prompt = “hello world”
sglang:
import openai
client = openai.Client(
base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.embeddings.create(
model="default",
input=prompt ,
)

transformer：
tokenizer = AutoTokenizer.from_pretrained('Alibaba-NLP/gte-Qwen2-7B-instruct', trust_remote_code=True)
model = AutoModel.from_pretrained('Alibaba-NLP/gte-Qwen2-7B-instruct', trust_remote_code=True)

max_length = 8192

batch_dict = tokenizer(prompt, max_length=max_length, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

embeddings = F.normalize(embeddings, p=2, dim=1)

[zhaochenyang20]

@llmforever hello. Sorry, I haven't noticed this before. Do you still need to fix this? Actually, we have a unit test for this in test/srt/models/test_embedding_models.py. The logits here is indeed closed.

Also, I don't understand what did you mean by "perform not so well". Could you provide your running snifts and your serving command for SGLang.

And, does e5-mistral also have this problem? Or only get?

[thomZ1]

The same problem, I tried using the SGLang OpenAI API and SentenceTransformer with the same prompt, but the output embeddings were different.

[zhaochenyang20]

Yeah. The embedding could be different due to a lot of reasons. @llmforever

You can check this unit test: https://github.com/sgl-project/sglang/blob/main/test/srt/models/test_embedding_models.py

We set a tolerance value for the embedding difference.

Also, please try the e5-mistral model and give us the embedding difference.

https://huggingface.co/intfloat/e5-mistral-7b-instruct

@Ying1123 Do you think the difference provided is tolerable?

[llmforever]

Yeah. The embedding could be different due to a lot of reasons. @llmforever

You can check this unit test: https://github.com/sgl-project/sglang/blob/main/test/srt/models/test_embedding_models.py

We set a tolerance value for the embedding difference.

Also, please try the e5-mistral model and give us the embedding difference.

https://huggingface.co/intfloat/e5-mistral-7b-instruct

@Ying1123 Do you think the difference provided is tolerable?

I test about 10 cases，each accuracy drop from 80% to less than 10%，i think the difference is not tolerable，but the result of the e5-mistral-7b-instruct model is the same，can you please help me look that? Here is the code i use to generate the embedding：

for transformer：

import torch
import torch.nn.functional as F

from torch import Tensor
from transformers import AutoTokenizer, AutoModel

input_texts = ['hello']
tokenizer = AutoTokenizer.from_pretrained('Alibaba-NLP/gte-Qwen2-7B-instruct', trust_remote_code=True)
model = AutoModel.from_pretrained('Alibaba-NLP/gte-Qwen2-7B-instruct', trust_remote_code=True)

max_length = 8192

batch_dict = tokenizer(input_texts, max_length=max_length, padding=True, truncation=True, return_tensors='pt')
outputs = model(**batch_dict)
embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])

embeddings = F.normalize(embeddings, p=2, dim=1)

for sglang:
import openai
import time
client = openai.Client(
base_url="http://localhost:30000/v1", api_key="EMPTY")

input_texts = ['hello']

queres = client.embeddings.create(
model="default",
input=quelist,
)
embeddings = torch.tensor(response.data[0].embedding)

[zhaochenyang20]

@Ying1123 I think he provides an intolerable difference hummm? I gonna check it these days.