Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/monkeypatch improvements #73

Merged
merged 2 commits into from
Dec 24, 2022
Merged

Feature/monkeypatch improvements #73

merged 2 commits into from
Dec 24, 2022

Conversation

hafriedlander
Copy link
Collaborator

Fixes #64

Also adds monkeypatch_or_replace_lora and monkeypatch_remove_lora (I see monkeypatch_or_replace_lora eventually replacing monkeypatch_lora and monkeypatch_replace_lora completely).

(Just re-raising #72 to be from a feature branch)

@cloneofsimo
Copy link
Owner

Whoa this is lot of work you've got there. Few questions ive got

  1. Was this backward compatible with previous checkpoints? Like the ones in root?
  2. Does this work with three training scripts?
  3. Does this work with CLIs?

@hafriedlander
Copy link
Collaborator Author

Most of this is whitespace changes, so it looks more than it is :).

There are three changes:

  • Add the helper functions _find_children and _find_modules
  • Change the monkeypatch (and other) functions to use those helper functions and
  • Add a new monkeypatch_or_update_lora and monkeypatch_remove_lora

So regarding your questions, it should be compatible with all of them because the structure doesn't change.* I've specifically tested with two of the three training scripts. I'll check the CLI in a bit.

(*The catch is that the out_v Lora parameters are now put in the right place, and if they had a bunch of junk in them for some reason that could cause problems. That hasn't been an issue for any of the ones I've tested though).

@cloneofsimo
Copy link
Owner

Awesome! Thank you so much for the contribution!

@cloneofsimo
Copy link
Owner

@hafriedlander Hi, Can you update me on the CLI's? Sorry to bother you. I should probably make a testing pipeline.

@hafriedlander
Copy link
Collaborator Author

@cloneofsimo yes, confirmed. I ran an lpl and upl merge, and the hash matched exactly for with both the code before and after this change.

(With one caveat: both required a change to cli, because you changed the default for unet to include "GEGLU" but the example lora_kiriko.pt doesn't include parameters for that Module. Using safetensors would fix that, since the safetensors files include what Layers they apply to.)

@cloneofsimo cloneofsimo merged commit 4869fe3 into cloneofsimo:develop Dec 24, 2022
@cloneofsimo cloneofsimo mentioned this pull request Dec 24, 2022
Merged
cloneofsimo added a commit that referenced this pull request Dec 25, 2022
@cloneofsimo @brian6091
@DavidePaglieri @hdon96 @hafriedlander
v 0.0.7 (#84)
66c18d3 
* Add parameter to control rank of decomposition (#28)

* ENH: allow controlling rank of approximation

* Training script accepts lora_rank

* feat : statefully monkeypatch different loras + example ipynb + readme

* Fix lora inject, added weight self apply lora (#39)

* Develop (#34)

* Add parameter to control rank of decomposition (#28)

* ENH: allow controlling rank of approximation

* Training script accepts lora_rank

* feat : statefully monkeypatch different loras + example ipynb + readme

Co-authored-by: brian6091 <brian6091@gmail.com>

* release : version 0.0.4, now able to tune rank, now add loras dynamically

* readme : add brain6091's discussions

* fix:inject lora in to_out module list

* feat: added weight self apply lora

* chore: add import copy

* fix: readded r

Co-authored-by: Simo Ryu <35953539+cloneofsimo@users.noreply.github.com>
Co-authored-by: brian6091 <brian6091@gmail.com>
Co-authored-by: SimoRyu <cloneofsimo@korea.ac.kr>

* Revert "Fix lora inject, added weight self apply lora (#39)" (#40)

This reverts commit fececf3.

* fix : rank bug in monkeypatch

* fix cli fix

* visualizatio on effect of LR

* Fix save_steps, max_train_steps, and logging (#45)

* v 0.0.5 (#42)

* Add parameter to control rank of decomposition (#28)

* ENH: allow controlling rank of approximation

* Training script accepts lora_rank

* feat : statefully monkeypatch different loras + example ipynb + readme

* Fix lora inject, added weight self apply lora (#39)

* Develop (#34)

* Add parameter to control rank of decomposition (#28)

* ENH: allow controlling rank of approximation

* Training script accepts lora_rank

* feat : statefully monkeypatch different loras + example ipynb + readme

Co-authored-by: brian6091 <brian6091@gmail.com>

* release : version 0.0.4, now able to tune rank, now add loras dynamically

* readme : add brain6091's discussions

* fix:inject lora in to_out module list

* feat: added weight self apply lora

* chore: add import copy

* fix: readded r

Co-authored-by: Simo Ryu <35953539+cloneofsimo@users.noreply.github.com>
Co-authored-by: brian6091 <brian6091@gmail.com>
Co-authored-by: SimoRyu <cloneofsimo@korea.ac.kr>

* Revert "Fix lora inject, added weight self apply lora (#39)" (#40)

This reverts commit fececf3.

* fix : rank bug in monkeypatch

* fix cli fix

* visualizatio on effect of LR

Co-authored-by: brian6091 <brian6091@gmail.com>
Co-authored-by: Davide Paglieri <paglieridavide@gmail.com>

* Fix save_steps, max_train_steps, and logging

Corrected indenting so checking save_steps, max_train_steps, and updating logs are performed every step instead at the end of an epoch.

Co-authored-by: Simo Ryu <35953539+cloneofsimo@users.noreply.github.com>
Co-authored-by: brian6091 <brian6091@gmail.com>
Co-authored-by: Davide Paglieri <paglieridavide@gmail.com>

* Enable resuming (#52)

* v 0.0.5 (#42)

* Add parameter to control rank of decomposition (#28)

* ENH: allow controlling rank of approximation

* Training script accepts lora_rank

* feat : statefully monkeypatch different loras + example ipynb + readme

* Fix lora inject, added weight self apply lora (#39)

* Develop (#34)

* Add parameter to control rank of decomposition (#28)

* ENH: allow controlling rank of approximation

* Training script accepts lora_rank

* feat : statefully monkeypatch different loras + example ipynb + readme

Co-authored-by: brian6091 <brian6091@gmail.com>

* release : version 0.0.4, now able to tune rank, now add loras dynamically

* readme : add brain6091's discussions

* fix:inject lora in to_out module list

* feat: added weight self apply lora

* chore: add import copy

* fix: readded r

Co-authored-by: Simo Ryu <35953539+cloneofsimo@users.noreply.github.com>
Co-authored-by: brian6091 <brian6091@gmail.com>
Co-authored-by: SimoRyu <cloneofsimo@korea.ac.kr>

* Revert "Fix lora inject, added weight self apply lora (#39)" (#40)

This reverts commit fececf3.

* fix : rank bug in monkeypatch

* fix cli fix

* visualizatio on effect of LR

Co-authored-by: brian6091 <brian6091@gmail.com>
Co-authored-by: Davide Paglieri <paglieridavide@gmail.com>

* Enable resume training unet/text encoder (#48)

* Enable resume training unet/text encoder

New flags --resume_text_encoder --resume_unet accept the paths to .pt files to resume.
Make sure to change the output directory from the previous training session, or else .pt files will be overwritten since training does not resume from previous global step.

* Load weights from .pt with inject_trainable_lora

Adds new loras argument to inject_trainable_lora function which accepts path to a .pt file containing previously trained weights.

Co-authored-by: Simo Ryu <35953539+cloneofsimo@users.noreply.github.com>
Co-authored-by: brian6091 <brian6091@gmail.com>
Co-authored-by: Davide Paglieri <paglieridavide@gmail.com>

* feat : low-rank pivotal tuning

* feat :  pivotal tuning

* v 0.0.6

* Learning rate switching & fix indent (#57)

* Learning rate switching & fix indent

Make learning rates switch from training textual inversion to unet/text encoder after unfreeze_lora_step.
I think this is how it was explained in the paper linked(?)

Either way, it might be useful to add another parameter to activate unet/text encoder training at a certain step instead of at unfreeze_lora_step.
This would let the user have more control.

Also fix indenting to make save_steps and logging work properly.

* Fix indent

fix accelerator.wait_for_everyone() indent according to original dreambooth training

* Re:Fix indent (#58)

Fix indenting of accelerator.wait_for_everyone()
according to original dreambooth training

* ff now training default

* feat : dataset

* feat : utils to back training

* readme : more contents. citations, etc.

* fix : weight init

* Feature/monkeypatch improvements (#73)

* Refactor module replacement to work with nested Linears

* Make monkeypatch_remove_lora remove all LoraInjectedLinear instances

* Turn off resizing images with --resize=False (#71)

* Make image resize optional with --resize

Toggle off image resizing using --resize=False. Default is true for to maintain consistent operation.

* Make image resize optional with --resize

Toggle off image resizing using --resize=False. Default is true for to maintain consistent operation.

* Make image resize optional with --resize

Toggle off image resizing using --resize=False. Default is true for to maintain consistent operation.

* Revert "Turn off resizing images with --resize=False (#71)" (#77)

This reverts commit 39affb7.

* Use safetensors to store Loras (#74)

* Add safetensors supports

* Add some documentation for the safetensors load and save methods

* Fix typing-related syntax errors in Python < 3.10 introduced in recent refactor (#79)

* Fix the --resize=False option (#81)

* Make image resize optional with --resize

Toggle off image resizing using --resize=False. Default is true for to maintain consistent operation.

* Make image resize optional with --resize

Toggle off image resizing using --resize=False. Default is true for to maintain consistent operation.

* Make image resize optional with --resize

Toggle off image resizing using --resize=False. Default is true for to maintain consistent operation.

* Fix resize==False functionality

* Update train_lora_pt_caption.py

* Update train_lora_w_ti.py

* Pivotal Tuning with hackable training code for CLI (#83)

* feat : save utils on lora

* fix : stochastic attribute

* feat : cleaner training code

* fix : bit of bugs on inspect and trainer

* fix : moved pti training to cli

* feat : patch now accepts target arg

* fix : gelu in target

* fix : gradient being way too large : autocast was the problem

* fix : hflip

* fix : example running well!

* merge master

Co-authored-by: brian6091 <brian6091@gmail.com>
Co-authored-by: Davide Paglieri <paglieridavide@gmail.com>
Co-authored-by: hdeezy <82070413+hdeezy@users.noreply.github.com>
Co-authored-by: Hamish Friedlander <hafriedlander@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hafriedlander @cloneofsimo