feat: Refactor post-processing of adapters #345
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
|
Thanks for making a pull request! 😃 |
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
Ssukriti
changed the title
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
Ssukriti
added a commit
that referenced
this pull request
Sep 23, 2024
* refactor saving tokens metadata Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * remove extra check Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * post processing script Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * post processing script Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix: unit test args Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * undo post_process_vLLm flag Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> --------- Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
Ssukriti
added a commit
that referenced
this pull request
Sep 25, 2024
* utilities to post process checkpoint for LoRA Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * improve code comments Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * Add unit test and fix some lint errors Signed-off-by: Angel Luu <angel.luu@us.ibm.com> * lint: fix more fmt errors Signed-off-by: Angel Luu <angel.luu@us.ibm.com> * feat: Add post_process_vLLM_adapters_new_tokens function to main Signed-off-by: Will Johnson <mwjohnson728@gmail.com> * fmt Signed-off-by: Will Johnson <mwjohnson728@gmail.com> * fix: Add post processing flag so post processing is only done for vLLM Signed-off-by: Will Johnson <mwjohnson728@gmail.com> * fix: get num_added_tokens from resize function (#344) * get num_added_tokens Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * remove extra code Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> --------- Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * Ran fmt and also removed unneccessary files from test artifact Signed-off-by: Angel Luu <angel.luu@us.ibm.com> * fix: unit tests Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix: Adding tokens in special_tokens_dict Signed-off-by: Abhishek <maurya.abhishek@ibm.com> * fix: Add additional arg to tests to reflect new flag post_process_vllm Signed-off-by: Will Johnson <mwjohnson728@gmail.com> * fmt Signed-off-by: Will Johnson <mwjohnson728@gmail.com> * feat: Refactor post-processing of adapters (#345) * refactor saving tokens metadata Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * remove extra check Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * post processing script Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * post processing script Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix: unit test args Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * undo post_process_vLLm flag Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> --------- Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * add test for LoRA tuning from main Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix formatting Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * correcting post processing script Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix:post-process in place Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * update documentation for post-processing Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix:formatting Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix:linting Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * more warnings /exceptions in script Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * check for no tokens added Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix:linting Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * additional unit test Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * add more tests Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix:tokenizer test Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix:linting and docstrings Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix:return type of trainer Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * test: enable tests and fix copytree Signed-off-by: Anh Uong <anh.uong@ibm.com> * use copy function from build Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * fix:linting and formatting Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * make build a module Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> * add back old copy function Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> --------- Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com> Signed-off-by: Angel Luu <angel.luu@us.ibm.com> Signed-off-by: Will Johnson <mwjohnson728@gmail.com> Signed-off-by: Abhishek <maurya.abhishek@ibm.com> Signed-off-by: Anh Uong <anh.uong@ibm.com> Co-authored-by: Angel Luu <angel.luu@us.ibm.com> Co-authored-by: Will Johnson <mwjohnson728@gmail.com> Co-authored-by: Abhishek <maurya.abhishek@ibm.com> Co-authored-by: Anh Uong <anh.uong@ibm.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of the change
Related issue number
How to verify the PR
Was the PR tested