-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
exclude_modules to keep specific layers or other quirky components out of a target_modules selection #2044
Comments
Thanks for opening this feature request. You're not the first one who would like to have this feature. If you or someone else wants to work on the PR, contributions are welcome. That said, I just wanted to ask if you know that there are already ways to achieve this. One way is to use The most flexible way to tackle this type of problem is, however, to pass a string to |
oh, i see. so you are suggesting i programmatically search the keys and build a list of targets that match my exclude and target patterns! that is pretty good. maybe it's enough |
What I meant is something like this: Say I have config = LoraConfig(target_modules=r".*\.(?!11)\d+\.fc1$") |
Hi @BenjaminBossan just trying to understand and if possible would like to take this up and contribute. The issue is to add an extra exclude params in the lora config to not use specific layers right? Is this needed since passing a 'fc' layers in target modules matches across all the layers? |
Yes, I think you've got it. If I understand the original request correctly, the use case is that we have N layers, each of which have |
alright @BenjaminBossan thanks, id like to take this up |
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. |
Resolved via #2102. At the moment, using this requires installing PEFT from source. |
Allows to exclude target modules.
Feature request
The LoraConfig class should accept an optional
exclude_modules
list of regular expressions in an analogue totarget_modules
that then should be referenced when matching against entries fromtarget_modules
.Motivation
Targeting
proj_out
andproj_mlp
on some models greatly improves the results, but we find that the effect is more reliable and robust if the final proj_out and proj_mlp layers are excluded.Your contribution
I can quickly integrate, test, and make use of this feature in bghira/simpletuner
I'm not sure I'll have time to submit a PR in the next two weeks, but after that perhaps I will be able to. Someone who grabs it before then will be much appreciated 💌
The text was updated successfully, but these errors were encountered: