Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: BERT models for embeddings #5179

Closed
mevince opened this issue Jun 1, 2024 · 4 comments · Fixed by #9056
Closed

[Feature]: BERT models for embeddings #5179

mevince opened this issue Jun 1, 2024 · 4 comments · Fixed by #9056
Labels
good first issue Good for newcomers new model Requests to new models

Comments

@mevince
Copy link

mevince commented Jun 1, 2024

Now with the introduction of embeddings: #3734, are there plans on the roadmap to support BERT models?

@robertgshaw2-redhat
Copy link
Collaborator

Yep - we would welcome a PR

@DarkLight1337 DarkLight1337 added good first issue Good for newcomers new model Requests to new models and removed feature request labels Jun 2, 2024
@Etelis
Copy link
Contributor

Etelis commented Jun 7, 2024

Sounds fun! I'm on this
Any notes or hints on that one?
@DarkLight1337
@robertgshaw2-neuralmagic

Thanks!

@laishzh
Copy link
Contributor

laishzh commented Jun 7, 2024

I also have interest in this task. Referring to #3734, I just use transformers.BertModel to implement the BertEmbeddingModel class(main...laishzh:vllm:feat/bert). The code is in very early version, but it can output the embedding which I think is wrong~ The reason maybe is that the weights are not loaded correctly. This is my first development. I'm not sure whether is the right way to implement, or need to reimplement BertModel? Suggestions or cooperation are welcome.

@Etelis Also hope it helps.

@robertgshaw2-redhat
Copy link
Collaborator

The main thing you have to do is implement the BERTModel or XLMRobertaModel in the vllm/model_executor/models directory using the layers in vllm/model_exeuctor/layers. And then register the model in the Registry.

You can look at how llama and others are implemented in that directory as inspiration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers new model Requests to new models
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants