Skip to content

Commit

Permalink
Only enable R4 as default setting
Browse files Browse the repository at this point in the history
Random R1 and R2 matrices are showing better results than just using R4, so the latter seems to be a better default option (at least for now)
  • Loading branch information
tobiasvanderwerff committed Oct 9, 2024
1 parent 77f11c3 commit 0bf6a76
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion torchao/prototype/spinquant/spinquant.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ def forward(self, x):
return x


def apply_spinquant(model: Transformer, use_r1=False, use_r2=True, use_r4=True, pretrained_rotation_path=None):
def apply_spinquant(model: Transformer, use_r1=False, use_r2=False, use_r4=True, pretrained_rotation_path=None):
"""
Apply SpinQuant to a Transformer model: https://arxiv.org/abs/2405.16406
Expand Down

0 comments on commit 0bf6a76

Please sign in to comment.