Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MNTP training #15

Merged
merged 11 commits into from
Apr 17, 2024
Merged

MNTP training #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
9ff4123
first iteration of script ,TODO: save properly
vaibhavad Apr 16, 2024
e920e61
model class loading
vaibhavad Apr 16, 2024
acd6617
bf16 flash attn
vaibhavad Apr 16, 2024
5d681fc
ignore wandb
vaibhavad Apr 16, 2024
c2296ac
added dependencies for running mlm training
vaibhavad Apr 16, 2024
67d9164
revert custom local change
vaibhavad Apr 16, 2024
2a08e1e
peft loading and saving
vaibhavad Apr 17, 2024
267299d
remove unused functions
vaibhavad Apr 17, 2024
81600a8
all configs
vaibhavad Apr 17, 2024
55cf489
minor formatting changes
vaibhavad Apr 17, 2024
b9b6803
flash attention needs no-build-isolation
vaibhavad Apr 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
build/
dist/
*.egg-info
**/__pycache__
**/__pycache__
wandb/**
output/**
6 changes: 4 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,17 @@ LLM2Vec is a simple recipe to convert decoder-only LLMs into text encoders. It c
</p>

## Installation
To use LLM2Vec, first install the llm2vec package from PyPI.
To use LLM2Vec, first install the llm2vec package from PyPI, followed by installing flash-attention:

```bash
pip install llm2vec
pip install flash-attn --no-build-isolation
```
You can also directly install it from our code by cloning the repository and:
You can also directly install the latest version of llm2vec by cloning the repository:

```bash
pip install -e .
pip install flash-attn --no-build-isolation
```

## Getting Started
Expand Down
Loading