Skip to content

Commit

Permalink
Add means to not load gpt2 wpe
Browse files Browse the repository at this point in the history
This will allow finetuning of other position embeddings for gpt2
variations.
  • Loading branch information
gkielian committed Sep 17, 2024
1 parent f0890e1 commit beff902
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions model.py
Original file line number Diff line number Diff line change
Expand Up @@ -890,6 +890,10 @@ def from_pretrained(cls, config, model_type):
if key == "lm_head.weight":
continue

if not config.use_abs_pos_embeddings:
if key == "transformer.wpe.weight":
continue

assert sd_hf[key].shape == sd[key].shape
with torch.no_grad():
print(key)
Expand Down

0 comments on commit beff902

Please sign in to comment.